Fast and Modular Autonomy Software for Autonomous Racing Vehicles (2024)

3.2.1 Challenges and Requirements

The AV-21 is capable of very high accelerations (greater than 20$m/s^{2}$) and speeds (greater than 180$mph$), meaning LiDAR or camera frame-to-frame movement can be significant. Additionally, there are no existing data sets for detecting AV-21s and little data showing the performance of sensors, such as LiDARs and cameras, at our target speeds. As a result, initially, data-driven approaches were not feasible, and the performance at higher speeds could not be immediately evaluated.

Initial efforts: Due to the lack of training data, our initial LiDAR perception approach for Season One utilized an unsupervised clustering algorithm. A Cloth Simulation Filter (CSF) [csf] was used to remove ground points and a density-based clustering technique, DBSCAN [dbscan], was used to identify obstacles of specific dimensions within the bounds of the track. While seemingly viable, this approach proved to have many failure cases, such as identifying dust above the track as an obstacle, shown in Figure 8. By observing these failures and the potential to break down at higher vehicle speeds, the need for a robust, efficient, learning-based perception approach was evident.

3.2.2 Overview of Approach

Final perception stack: Figure 9 shows our final perception stack’s decoupled and multi-modal approach to accurately detecting and localizing all other agents on the track, which notably, no longer makes use of clustering. On the AV-21 Platform (Figure 5), there is an assortment of LiDARs, cameras, and radars, each with advantages and disadvantages. For example, cameras alone do not give an accurate depth estimation but can operate at a much higher frame rate (up to $75Hz$) and resolution ($2064\times 960$) than LiDARs. A camera-based detection pipeline can provide a higher frequency update on our belief of the world and has the potential to see other agents from further away. Figure 10 shows some additional challenges faced with perception. To best exploit the sensors’ strengths and for robustness and redundancy, our perception stack uses each sensor independently and in parallel and feeds all localized detections to our tracking stack.

3.2.3 Camera

3.2.4 PointPillars

The AV-21 platform has three Luminar Hydra LiDARs[luminarhydra] positioned in a triangular fashion. Each LiDAR has a field of view (FOV) of $120^{\circ}$, together allowing for $360^{\circ}$ coverage around the vehicle. Each LiDAR is capable of excellent coverage at over $100m$, thereby providing an over $200m$ radius circle of coverage around the track. Since the track is only so wide, this cloud is cropped further to being $200m\times 40m$. An example cloud can be seen in Figure 12.

Numerous Deep Learning methods of object detection using LiDARs have shown promising results, such as VoxelNet [voxelnet], PointRCNN [pointrcnn], SECOND [second], and others. Low-latency inference and accurate detections are of the utmost importance for our use case of high-speed autonomous racing. For this reason, PointPillars [pointpillars] serves as our primary detection method, capable of reliably detecting vehicles at ranges up to $100m$ away. The birds-eye-view projection and 2D convolutions used within PointPillars allow for the removal of computationally expensive and time-consuming sparse 3D convolutions performed by other LiDAR networks.

Figure 13 shows the outcomes of PointPillars. To reduce processing time, our PointPillars implementation is single-sweep, meaning we do not accumulate scans over time before running inference. Additionally, to further simplify the pipeline, inference is done directly on the raw scans, after down-sampling and applying a crop. We explicitly chose to not compensate for distortion caused by the ego vehicle’s motion. Based on the data observed and practical considerations within the larger stack, motion compensation was deemed not worth the additionally complexity and processing time required. The scanning rate ($\sim 50ms$ from top to bottom) is faster than other LiDARs, which results in less distortion. Additionally, the LiDAR data is only used for detection and the relative speed between agents is low enough that the error due to motion distortion can be ignored. The potential gains do not outweigh the additional latency. Finally, work had been done to implement distortion correction, but was removed due to integration and performance issues, which will be discussed further in Section 4.

Our implementation of PointPillars uses many of the same underlying optimizations seen in [BEV_FUSION], such as heavy utilization of Torch Sparse [tang2022torchsparse], a high-performance neural network library, specializing in point cloud processing. This allows for fast and efficient inference time, even with a Python-based implementation.

3.2.5 Data Collection, Labeling, & Training

No dataset exists that contains AV-21s racing head-to-head. Adequately training PointPillars required developing a large and robust dataset. Initially, data was collected in simulation, which helped developed an initial model. The first model dataset is broken down in Table 2. The simulation environment did not match the vehicle setup perfectly. In particular, while the range and coverage were similar, the point cloud was less dense than in real life. Interestingly, we found that the initial model trained off of this data transferred to detecting AV-21s on real data, especially at longer ranges, where the cloud is less dense. Figure 14 shows a comparison of PointPillars detections against the measured trajectory of an opponent ARV.

With an initial model, it was now possible to do “auto-labeling”, where the model is used to generate new labels that are then hand-verified by a human annotator. Because the model often provides a detection that is close to ground truth, the workload on the human annotator is reduced. Additionally, by using the existing model to label more data, labels can be focused on the areas where the model performed most poorly.

3.2.6 Discussion: Strengths, Limitations, and Future Work

Given the prototypical nature of the AV-21 platform, the sensor plate must be disassembled every time the autonomy components need servicing. As a result, the extrinsic calibration between the cameras and the LiDARs changes frequently. This is less of an issue with the LiDARs, as they are all firmly fastened to the same aluminum plate. Additionally, the extreme operating conditions of the AV-21 platform (i.e. high speeds and accelerations) also necessitate re-calibrating the sensors regularly, even if the sensor plate has not been removed. Small extrinsic calibration errors can lead to very large projection errors, especially for distant objects. This problem is not exclusive to ARVs and is an active area of research [aurora_calibration][calibration_survey]. Future work will center on streamlining the calibration process and developing systems that are less brittle to small errors.

Finally, due to a severe crash less than 72 hours before the competition in Season Two at Las Vegas, the camera detection pipeline was disabled for the competition events. An image from the footage of the crash, recorded by an onboard GoPro camera, can be seen in Figure 15. With the focus being repairing the AV-21 vehicle, no time was available to properly calibrate the sensors and the potential for projection errors outweighed the benefits. Because of the modular design of the perception stack, it was trivial to make such a drastic change. In a fast-paced competition environment, this modularity and flexibility proved paramount in allowing the vehicle to operate during the competition. While the redundancy and peak performance of the stack was compromised, as shown later in Section 4, the vehicle was still able to autonomously compete in three rounds, winning the first two, and losing the third after running out of fuel after attempting an overtake at over $150mph$.

3.3.1 Challenges and Requirements

Tracking within an ARV software stack serves to provide downstream tasks with a single belief of the states of other agents within the world. Different perception modalities capture different portions of a given agent’s state space. For example, the monocular camera perception provides a noisy estimate of an agent’s position, but cannot accurately predict its orientation. Our LiDAR perception produces full pose estimates of other agents, but currently does not infer the agent’s velocity. While using only one of these detection methods will yield a belief that is severely limited by the outlined weaknesses, the effective fusion of both can result in each modality compensating for the drawbacks of the other.

Our implementation allows for the fusion of multiple sensing modalities in a straightforward manner, and serves to provide downstream planning tasks with the state of all perceived agents. Our decoupled approach to perception requires our tracking stack to meet the following requirements:

Finally, tracking must perform all of the above while ensuring as little additional latency as possible, handling measurements from perception asynchronously and out of order, and compensating for any delay between sensor measurement and tracking.

3.3.2 Overview of Approach

Our approach consists of three main components: Filtering, Association, and Fusion. Filtering removes outliers. Association determines whether or not a detection is of a previously seen agent. Fusion is incorporating new measurements of agents’ states. In order to minimize processing latency within the tracking stack, well-researched, efficient algorithms are leveraged for each module. Figure 16 presents the Tracking pipeline architecture.

Filtering: Detections filtered by a confidence threshold, a hyper-parameter within our tracking stack, tuned empirically by analyzing the false positives and associated confidence produced by perception. Additionally, any detection that falls outside of the track bounds is ignored. The combination of these two filtering steps helps to ensure that only valid detections are processed and used to generate tracked agents.

Association: AB3DMOT [AB3DMOT] provides the data association module utilized by tracking stack. By employing two computationally efficient algorithms, the Hungarian algorithm for data matching [Hungarian] and the Kalman filter [KF] for fusion and prediction, the authors demonstrate strong results on multiple open-source data sets while also providing high-frequency predictions. In practice, we observed that the Hungarian algorithm with Euclidean distance often resulted in poor data association, especially during temporary sensor drop-out. Therefore, our implementation uses simple greedy matching with the Mahalanobis distance[Mahalanobis], which performed better in testing.

Track births and deaths: To reduce the probability of false positives becoming valid tracks, a new potential track is instantiated (born) only after two detections (from successive sweeps) are associated with it. This hyper-parameter provides a means to balance between the quality and confidence of tracks and end to end latency in reacting to other agents. Finally, any tracks that have not had a detection associated with it within the last five seconds are also removed (killed) to prevent stale tracks from influencing future associations.

Fusion: Once detections have been associated with an existing tracked agent, or have been repeatedly observed and classified as a new agent, we begin tracking the agent using fused multi-modal perception outputs. Again, we utilize a modified version of [AB3DMOT] as the Kalman filter for performing sensor fusion. Since incoming detections from the camera perception pipeline have already been projected into a 3-dimensional position and transformed into a common frame, both LiDAR and camera measurements can be used to update the internal Kalman filter for a given tracked agent. In this way, sensor fusion becomes a simple task that can be asynchronous across the two modalities, and the states of tracked agents can be published at the receipt of each incoming detection.

3.3.3 Discussion, Limitations, and Future Work

The tracking pipeline meets all requirements and is sufficiently accurately and performant to handle the IAC Passing Competition. Our modular design was especially important when the camera perception was disabled on race day, outlined in detail in Section 3.2.6. Given these successes, however, our system has not been robustly tested against multiple agents, specifically agents that are close together (i.e. $\leq 5m$). In traditional motorsports, humans drive aggressively in close proximity to one another. While the competition format is far from this style of racing, future works will need to handle such operating domains in order to challenge professional drivers. Multiple agents in close proximity are more difficult to track, due to higher association ambiguity and occlusions.

Finally, a Kalman filter will be replaced by an Extended Kalman filter (EKF) to enable a non-linear motion model. With an EKF and a better motion model (i.e. constant curvature), predictions of agent tracks will be more accurate, which is especially important during periods of infrequent detections (i.e. the other agent is in the blind-spot produced by the rear wing of the vehicle).

3.4.1 Overview

We developed a fast, modular motion planning stack capable of predicting agent behavior and safely trailing and passing other agents. Figure 17 shows a high-level overview of the motion planner. The path planner identifies the agent’s position at points in the track where it is most optimal to pass and extrapolates its position in the following lap. We have based our path planning approach on a set of primitive behaviors from which higher-level strategies can select. These primitive behaviors include opponent trailing, raceline following, and lane switching. This modularity allows for a clean separation between high-level decision-making and trajectory selection and generation.

We also take advantage of knowing the track geometry and develop a set of strong priors in the form of a trajectory bank. Offline, trajectories are generated for various lanes along the track, providing different levels of clearance from the inner and outer track boundaries. Online, the planner selects the best trajectory from the bank based on the selected action primitive.

3.4.2 Offline Trajectory Generation

The raceline generation is a two-step process that begins with defining a set of waypoints on the track. Afterward, splines are interpolated on these waypoints to ensure continuity. This process relies heavily on the manual selection of waypoints at apexes to properly leverage spline properties.

For the overtaking competition, waypoints were selected manually from two lanes. We consider a lane to be defined as a path with both an inner and outer boundary and derive a center-line equidistant from the two boundaries. We obtain these two lanes by dividing the width of the track in half along the full length. Within these lanes, the process of manual sampling begins. A series of $N$ waypoints, $(q_{1},...,q_{N})$, are used to interpolate closed Spiro splines [Levien09]. Figure 18 displays waypoint selection and the interpolated raceline for the inner lane.

While the tracking controller is compatible with any reasonably smooth raceline, the Spiro spline family is well-suited for several reasons. Firstly, Spiro splines maintain $G^{4}$-continuity, meaning that the second derivative of curvature is continuous. In addition, Spiro splines are an efficient approximation of the Minimum Variation Curve (MVC), which minimizes the integral of curvature rate [Levien09], corresponding to minimizing the steering effort of the vehicle. By selecting waypoints at apexes, the above properties make a raceline generated from a Spiro spline effective at minimizing downstream instability, reducing steering effort, and resulting in faster lap times and smoother driving than naively selecting waypoints alone. Future work includes incorporating the Spiro spline representation within a larger optimization, such as the work in [tum_min_curv] and [tum_min_time], to eliminate the need for manual waypoint selection and ensure time-optimal trajectories.

3.4.3 Online Action Selection

The multi-agent passing competition sets two roles for the competitors: ”defender” and ”attacker”. As defined in Section 2, the attacker passes the defender that is maintaining a raceline within the inside lane of the track (see Figure 4). The attacker is also responsible for maintaining a safe distance from the defender. We define a set of Action Primitives to encode these maneuvers:

Selection of the primitives is dependent on the current role (defender or attacker) and the current Track Condition (i.e., Green, Yellow, Red, Waving Green), which are both defined by flags sent to the vehicle from Race Control.

While this logic is simple for the passing competition, the use of these primitives can scale to more complex logic. For example, rewards and costs could be assigned to each of these primitives which are then utilized by a search-based planner or a reinforcement-learning algorithm to estimate expected rewards and costs by taking a set of actions and to determine the best solution given the scenario. By structuring the planner around a modular set of action primitives, we can explore multiple solutions to the behavioral decision-making problem very easily. Additionally, it is easy to add more primitives in the future when necessary.

3.4.4 Online Raceline Merging

Merging between inner and outer racelines (i.e. “Safe merging”) is accomplished by first calculating the closest pose on the inner raceline $q_{k}^{in}$ corresponding to every pose on the outer raceline $q_{k}^{out}$. Next, given a start and a final interpolation pose, $q_{k}^{in}$ and $q_{k}^{out}$, as well as the corresponding twist, the spline optimization formulation in Section 3.2 of [Veerapaneni_minjerk] is used to generate a minimum jerk merging trajectory. Time intervals $h_{k}$ between consecutive time stamps can be decreased as desired to generate a sufficiently smooth raceline between $q_{k}^{in}$ and $q_{k}^{out}$. This interpolation can be seen in Figure 19.

3.4.5 Results and Discussion

Figure 17 displays a comprehensive flowchart of the mentioned approach and highlights the offline and online computations. Tasks performed by individual components of the planner are agnostic to the actions performed by other components. This includes adding or removing additional raceline primitives as well as action primitives, making this architecture highly modular. Future work on the planner includes automating the process of waypoint selection based on given inside and outside lines, which would consider varying track curvature. Finally, this approach is very fast, efficient, and able to easily execute at $20Hz$ (and capable of a much higher execution rate). Action selection, safe-merge trajectory interpolation, and velocity profiling are all very inexpensive to compute online.

3.5.1 Overview of Approach

Our controls architecture was designed for speed and modularity. To achieve this, we decompose our control task of path tracking into three tasks - lateral control, longitudinal control, and gear selection. Lateral tracking generates a steering angle such that the vehicle converges to the path. The longitudinal controller maintains a particular longitudinal velocity. The gear-shifting controller is responsible for selecting the optimal gear for maintaining the current vehicle speed. The majority of the content in this section will focus on the lateral tracking element of the vehicle’s controller. The architecture of the stack is described in Figure 20. Finally, the work presented here is a continuation of our previous work in [ICRA_WS_CONTROL].

Longitudinal Control and Gear ShiftingThe throttle and brake control utilizes a simple P control scheme paired with a feed-forward term to account for drag. This is expressed within Algorithm 1 line 1, which returns a command value. This command is then interpreted as either a throttle or brake command depending on whether the command is positive or negative, using a scaling factor on the brake signal to account for differences in magnitude between the throttle and brake. Smoothing on the throttle and brake signals prevents instantaneous acceleration or deceleration to help prevent vehicle instability. Algorithm 1 summarizes this algorithm.

The gear-shifting strategy is based on a simple lookup table where the optimal gear is computed given the current velocity of the vehicle. This gearing table was generated based on a model of the engine and track parameters to maximize torque, the computation of which is outside the scope of this paper.

Lateral LQR ControlThe lateral controller is built around a Linear-Quadratic Regulator (LQR) that generates an optimal feedback policy based on a nominal vehicle dynamics model. LQR was chosen because (1) it is inexpensive and fast to compute the desired controls online and (2) it is optimal given the vehicle dynamics model. However, one downside is that LQR does not reason about actuation constraints and only considers the error concerning the reference at the current time step. However, the vehicle primarily drives on an oval race track where only a few degrees of steering are required even for the most aggressive maneuvers. Hence, LQR can stably control the vehicle and meet the reference tracking requirements to safely avoid the other agent, track barriers, and maintain the desired trajectory.

Given a continuous-time linear system of the form in Equation 2 a quadratic cost function such as Equation 3 can be defined which is constrained by Equation 2. Equation 3 also has the constraints that $Q=Q^{T}\geq 0$ and $R=R^{T}\geq 0$.

Dynamic Bicycle ModelWe utilize a four-state dynamic bicycle model from [Rajamani_2005], shown in Equations 5 and 7. As noted in Section 3.5.1, we decompose the problem into separate lateral and longitudinal controllers. The lateral controller reasons about the vehicle’s lateral position $y$ and yaw angle $\psi$ in the Local Tangent Plane (LTP) frame provided by localization. The input for this system is the steering angle $\delta$. It also assumes some constant velocity $V_{x}$, which is the forward component of the speed ignoring lateral slip. The $C_{\alpha f}$ and $C_{\alpha r}$ parameters are the cornering stiffness of the front and rear tires respectively, which reflect the ability of the tires to resist deformation while the vehicle corners. In addition, $l_{f}$ and $l_{r}$ represent the length of the vehicle from the center of gravity to the front and rear axles respectively. The parameter $I_{z}$ is the scalar moment of inertia about the z-axis, and $m$ is the mass of the vehicle. The coordinate system is displayed in Figure 21 and vehicle parameters are summarized in Table 3.

With this, we can derive the error dynamics for which we solve the LQR problem to get the optimal stabilizing controller gain $K$ to drive the error to zero.

This model makes several assumptions, including small angle assumptions on the steering angle, which hold well on ellipsoidal tracks where we command a maximum steering angle of approximately $0.1$ [rad]. The model also assumes constant velocity. A reformulation to a linear time-varying model could allow for varying velocity. However, we accept that assumption and mitigate it with a series of feedback controllers derived at different velocities for simplicity. For greater detail on this formulation of the dynamics see [Rajamani_2005].

Pure Pursuit, LQR Tracking AlgorithmGiven the model formulation and the LQR formulation, we can make a lateral tracking algorithm by combining a pure-pursuit style look-ahead point and a feedback mechanism similar to Equation 4 generated using LQR. The look-ahead point is queried simply by looking for the point on the raceline provided by planning that is ahead of the vehicle a given distance $d$ away. LQR requires an $A$, $B$, $Q$, and $R$ matrix to generate the feedback policy. The $A$ and $B$ matrix can be obtained using the dynamics from Equations 5 and 7. The $Q$ and $R$ matrices can be obtained empirically where the diagonal $Q$ matrix defines the weights on the error terms in Equation 6, and the scalar $R$ matrix defines the weight on the steering angle that acts as a dampening factor to prevent over-steering.

A different parameter set $p=(Q^{v}_{(0,0)},Q^{v}_{(1,1)},Q^{v}_{(2,2)},Q^{v}_{(3,3)},R^{v}_{(0,0)})$ is utilized depending on the current velocity $v$ of the vehicle as shown in Table 4. Offline, the velocity bracket ranges $[\dot{x}_{\text{low}},\dot{x}_{\text{high}})$ are defined to cover the range of velocities the vehicle is expected to cover, or generally the range $[0,\infty)$ with no overlap between them.

The speed brackets serve two purposes: first, it provides linearization points across a range of velocities to account for the constant velocity assumption in the dynamics model. Secondly, as the vehicle reaches higher speeds, the expectation is that the dynamics model mismatch is greater. Additionally, it is preferred that the controller is less reactive at higher speeds. By varying the $R$ gain, we can tune the controls to be less aggressive as we reach higher speeds. By varying the $Q$ matrix, we can provide different emphases on the various components of our error, i.e. lateral & yaw deviations and their derivatives. Empirically, we have found that, at higher speeds, the vehicle is more stable when the yaw & yaw rate error consists of a higher overall emphasis in the resulting control output than the lateral deviations. Finally, the pure-pursuit look-ahead distance is also varied as a function of speed where $d=d_{\text{base}}+k_{v,d}\dot{x}$. These parameters provide numerous knobs for a controls engineer to rapidly tune the controller to achieve the desired performance.

The LQR tracking algorithm runs as follows. First, it initializes by loading in all velocity brackets and the associated parameters. It then generates the feedback policies $K$ for each bracket using the average velocity of the bracket. Note that if one of the bounds is $\infty$, the lower velocity is utilized. Next, at each time step, given the current state of the vehicle and a raceline, it performs a look-ahead query on the raceline to get the goal point. Using this goal point and the current state, an error state can be generated. Finally, with the current vehicle velocity, it queries the velocity brackets for the relevant feedback policy which is applied to the error state to get the optimal steering angle. This algorithm is summarized in Algorithm 2.

3.5.2 Results and Discussion

The controller stack shown in Figure 20 was evaluated over several performance laps at the LVMS with velocities ranging between $25m/s$ and $60.5m/s$. We experienced the worst performance at the highest speed targeting $60m/s$. We plot velocity and cross-track error (CTE) for the portion of a run in which speeds of $138mph$ were achieved in Figure 22. The maximum CTE experienced was $1.3m$, which occurred around bends, while the mean absolute CTE was $0.323m$. Additionally, the average CTE for several high-speed runs ($138mph$, $141mph$, and $137mph$ max speeds) at various speed brackets are presented in Table 5. Notably, for the run on January $7^{th}$, the controller was tuned to be less aggressive, which explains the degraded tracking.

We also evaluate the controller on lane-change tasks on the same track. Results from Season One are shown in Figures LABEL:fig:merge_out 23. Season Two results are show in Figure 24. The Season One results are from running the planner and controller in the loop on the vehicle, but the opposing agent was simulated, referred to as a “ghost” agent, as the vehicle will react to something that is not physically present. Results from the semi-finals match in Season Two, a real AV-21 multi-agent passing competition event, are also presented. Overall, the controller works very well in both seasons, with the CTE during the lane change maneuvers, staying between approximately $0.5-1.5m$. In the Season Two match, the vehicle reached lateral accelerations over $20m/s^{2}$.

3.6.1 Challenges

To localize itself on the track, the AV-21 is equipped with two dual-antenna NovAtel PwrPak7D-E1 GNSS units with RTK, as well as individual wheel speed sensors (Table 6). However, the high speed and high acceleration nature of the competition necessitated a custom solution that is robust to partial and total failures. In particular, a solution was needed that could meet the following requirements, all while traveling at high speeds:

3.6.2 Overview of Approach

The node responsible for all of the above is called the Localization Executive. When building the Localization Executive, it was important to have a flexible, generic, and modular framework for defining sources, safety thresholds, and defining safety checks and heuristics. As testing progressed and more track time was put onto the vehicle, we quickly learned and adapted our fusion strategies. For example, the quality of the NovAtel with its dual antennas on the top and front nose of the vehicle often produces an overall better heading and position result than the other unit, likely due to a wider baseline. The Localization Executive’s modularity allows us to prioritize high quality and filter out bad measurements on a per-source (i.e. pose, heading, velocity) basis. An example of a degraded source can be seen in Figure 26.

The resulting transformed and filtered measurements are then processed by Robot Localization [Rl_EKF], an open-source EKF package that provides a flexible and modular interface for fusing an arbitrary number of odometry sources. Robot Localization was chosen because 1) it is very well tested and maintained in the robotics and unmanned vehicles community, 2) its modular design provides extreme flexibility during development and testing, and 3) it works well and exposes several tuning parameters and control, while able to update its state at $100Hz$ and fuse all incoming measurements asynchronously, including with the ability to handle out of order measurements.

Finally, health status flags for the various sources and final odometry results are communicated to the System Executive, the high-level arbiter of the stack. If any of the following scenarios occur, the vehicle is brought to an immediate controlled stop:

If one unit is healthy, or if a combination of pose, heading, and velocity can be constructed from measurements from both units, then the stack will continue operating normally. Additionally, all health status flags are also communicated to the base station and presented in the interface, which is presented in Section 3.7. If partial, but not disastrous, failures occur, it is the responsibility of the base station operator decide whether to terminate the run or not.

3.7.1 Primary Interface

The primary interface displays track states, vehicle states and speeds, watchdog states, sensor frequencies, GPS health, and the computer’s compute utilization. Figure 28 shows the display for the primary interface, made up of a GUI using PyQT and an odometry display using RViz. Using PyQT, we can easily add or modify panel modules on the GUI to fit our area of concentration for a test run or competition run. With RViz, we can visualize the path the vehicle is following, the current vehicle odometry, and any agents we have detected and are tracking. RViz is easily configurable for what topics and data we want to see. This interface is meant primarily to be an information display and not to give base station operators control over vehicle functions.

Available telemetry information includes the state of the power train, vehicle position and speed, and target speed. This information informs the operator if the vehicle is in a healthy state overall and what is it trying to do. The CPU and memory usage is also displayed to ensure the computer onboard is running smoothly. The watchdogs serve as surveillance agents over the stack, indicating if any component of the stack fails or crashes. When a watchdog fails, the stack will intelligently bring the car to a safe stop. All sensor frequencies are displayed, indicating if there are any current or imminent problems within the perception or localization stacks. Colors vary based on different safety thresholds (i.e. “good”, “warning”, “bad”). GPS health information is also displayed.

3.7.2 Secondary Interface

The secondary interface displays all incoming telemetry data and more readings, such as fluid pressures and controller errors and commands. Figure 29 shows the display for the secondary interface. Like the primary interface, the secondary is modular and easily configurable to fit the area of concentration. It is constructed using PlotJuggler to plot all of the incoming telemetries. The interface serves to provide a detailed live feed of the health and performance of the vehicle and the stack, rather than serving as a quick-glance check. The benefit of the secondary interface is that operators can monitor the trends of the vehicle, and determine possible impending issues.

4.1.1 Run Objectives

The intended goal of the run was to perform a thorough verification of the controller’s performance to maintain a fixed speed and its ability to quickly and safely slow down in the event of a safety trigger, such as a communication or race control timeout. Because the track is severely banked (up to over $20^{\circ}$ in the steepest portions), too sudden of decelerations can result in a spin-out, as the rear tires lose grip with the ground during a deceleration event, due to a forward shift in the mass distribution of the vehicle. When the tires lose this grip, the vehicle is more prone to spinning out and losing control. So, the controller must maintain safe, smooth decelerations at all times.

After verifying the speed controller’s performance, the run was supposed to progress onto multi-agent testing; however, the second team was not ready and our sing-agent run continued. At this point, we set a goal to reach higher speeds (greater than $145mph$) and see the controller’s performance above our previous record of $138mph$. While we did not achieve our original goal, the insights gleaned from what ensued are likely valuable for future research.

4.1.2 Pre-Instability Performance

During the repeated braking events, the controller maintained stability. The speed controller still needed more tuning, and the purpose of this run was to collect data to further tune the controller. Notably, there was a point at about 700 seconds into the run where the braking resulted in some traction issues, which corresponds to the jump in cross-track error as the vehicle slows to a stop. Information like this helps provide an estimate of where the braking limits of the tires are. Overall, as noted in Table 7, the average cross-track error (CTE) over the entire run was $0.244m$. When traveling over $100mph$, the CTE was $0.501m$. At its worst, the CTE does not exceed roughly $1.25m$ when traveling around the corners (see Figure 30).

The G-G diagram plots the longitudinal and lateral accelerations. As tires have a maximum deformation, thereby maximum total force, it is important to evaluate what region the vehicle and tires are operating in. For example, if the vehicle were braking or accelerating into a corner, a high lateral and longitudinal acceleration is expected. However, if the vehicle is maintaining the same speed while traveling around the track, the expectation is that the vehicle would not see as high longitudinal accelerations, but still have significant lateral accelerations as it makes the turns.

In this run, due to the repeated speeding up and braking, the vehicle experienced a wide spread of accelerations. As shown later, the vehicle was accelerating significantly even while going into the corners, which is a precarious scenario because there is a risk of saturating the tires, which is ultimately what happened. Preventing future events like this will require explicitly modeling the tire limits and constraining the vehicle’s controls to stay within the dynamically feasible and safe region. However, determining these limits is still an open area of research.

4.1.3 Spin-Out

The night before January 3rd at LVMS saw the weather settle down to freezing temperatures. As it progressively turned colder, dipping to around $30^{\circ}$F by morning, conditions on the track became too cold to operate safely on. These conditions forced teams to shift testing operations to begin in the afternoon when temperatures reached about $50^{\circ}$F. The tires and power train of the AV-21 are designed to operate in warmer conditions. Freezing temperatures, such as those seen on January 3rd, can negatively impact performance.

The most obvious consequence of lower-than-optimal tire temperatures on the AV-21 is reduced tire traction and grip. In traditional motorsports, even when conditions are very warm, human drivers must ensure their tires are brought up to an optimum temperature. A race car’s tires must be operating at a temperature of at least $175^{\circ}$F to generate traction and grip [smith_tune_1978]. At the time of the spin-out, the tires were only at about $29^{\circ}$C, or about $85^{\circ}$F.

In Figure 31, the vehicle is stable up until the point of failure. Notably, the tires were very cold (about $84^{\circ}$F); meanwhile, the vehicle was commanded to speed up very aggressively (command increase of about $20mph$) while going into the corner. These commands were issued by the base station operator. Additionally, given how the rear wheel speeds spike while the front wheels do not, traction was lost on the rear tires, indicating an over-steering event. Finally, due to the decoupled longitudinal and lateral controls, the steering controller was not able to dictate a more reasonable velocity profile given the trajectory ahead. Instead, the base station operator was setting a speed, and the speed controller was ramping up to meet it, regardless of where the vehicle was on track and what trajectory it was taking.

4.1.4 Lessons Learned

Several insights and changes were made resulting from this incident, some software, others operational. First, the base station operator needs to verify that the tire temperatures are warm enough before trying to command the vehicle to a high speed. At the time of the incident, the base station did not display tire temperature. Moving forward, this feature has been added. Additionally, when doing constant and high-speed tests, unless there is confidence in the vehicle’s ability to navigate while maintaining that fixed speed, speed changes are avoided on turns.

Secondly, the development of improved tire modeling, with an online estimation component, is underway. It is not enough to set tire model parameters pulled from a data sheet and expect the vehicle to perform as expected all the time; rather, some online estimation of the tire parameters is needed. This is not a novel idea; rather, it has been explored before [tum-friction].

Finally, to take advantage of a better tire model, it is important to have a controller that can better reason about the vehicle dynamics and can jointly optimize the vehicle speed, steering, and accelerations. Development is underway on a model-predictive controller (MPC) that can predict and optimize the vehicle’s performance over a finite time horizon. With a better vehicle model, controller, and better operating procedures, spinouts should be less common.

In terms of total cost and damage, the spinout was relatively minor, with a single bent rear suspension piece. However, the data collected and insights are invaluable for future development and testing. One of the operating limits of the tires has been found, a limit that would not have been found otherwise. Because of the insights learned and data collected, this run was considered a success by the team, despite the result.

4.2.1 Perception & Tracking Failures

In the run-up to competition day, several of last-minute decisions and changes were made to the software, vehicle and sensor configuration, and computer hardware. All of these changes compounded and resulted in a total failure of our perception and tracking stack on the last day to qualify for the head-to-head competition event. While disappointing, the lessons learned reached far beyond software design and development.

4.2.2 Camera Hardware & Driver Changes

Due to issues with the network connections on the vehicle, camera drivers had to be reconfigured to reduce their network bandwidth, resulting in the streaming of wrongly-cropped or poor quality images. These images, examples shown in Figure 32, on top of a lack of training data from the LVMS track, resulted in a higher-than-expected false positive rate.

4.2.3 Failed Motion Compensation Changes

Since the LiDAR is not scanning every point instantaneously it is important to account for the ego vehicle’s motion during the LiDAR scan for the highest-accuracy result. However, leading up to the competition, the solution developed was not thoroughly validated on the vehicle while it was under full system load. On the AV-21, there are three LiDARs, each scanning at $20Hz$. The motion distortion node needed to combine all three clouds, project each point into the world to correct the distortion, and, optionally, project the cloud back into a local frame of reference. ROS 2 provides a message synchronization library that gives a convenient interface to subscribe to multiple topics and have all three topics delivered at the same time, in the same callback.

When under full system load, the DDS middleware and synchronization library was not delivering every LiDAR measurement from all three LiDARs. Messages were frequently being dropped. However, because the issue only showed up when the system was under load, it was not discovered until deployed and run on the vehicle during the qualification run.

Due to the dropped messages, the effective LiDAR frame rate received by PointPillars dropped to below $5Hz$. This corresponds to an average period of roughly $200ms$. Assuming sensor processing is keeping up with the sensor frame rates, the expected period is $50ms$, as the LiDAR is scanning at $20Hz$. In actuality, the period between LiDAR measurements was very inconsistent, often spiking over half a second (see Figure 33). This inconsistent and slow update rate compounded with other issues with the tracker that resulted in very poor perception performance overall.

4.2.4 Poorly Tuned & Flawed Outlier-Rejection

With the camera pipeline producing false positives, albeit at a low rate due to network bandwidth issues; and LiDAR running at a much lower frequency than what was expected, the fusion relied primarily on the Radar for tracks. However, last-minute code changes and tuning resulted in poor outlier rejection, which is especially important due to the high noise in radar tracks. While some good tracks surfaced and were followed, tracking failed to prune too many false positives. Additionally, the radar can only see in front of the vehicle, which is insufficient on its own for passing.

4.2.5 Lessons Learned

All three issues quickly compounded and it was clear to the base station operator that there were too many false positives from the perception and tracking stack. As a result, a strategic decision was made to terminate the run and have the vehicle return safely home. While a disappointing result, the run provided invaluable data and insights into testing and deploying a full, complicated system onto real hardware. Most importantly, this failure highlighted the importance of proper integration testing to identify unexpected issues sooner. Additionally, the next section will address the solutions that were developed and tested over Season Two and the resulting performance on track in competition.