You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-23 - 09:24

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Mon Mar 23, 2026 00:45:09)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2460299581,5cyclictest0-21swapper/119:20:151
2460299533,39cyclictest6217-21ssh22:20:121
2461099521,28cyclictest9732-21ssh22:40:132
2459499521,4cyclictest131rcu_preempt22:15:130
2461099512,26cyclictest14065-21ssh23:55:132
2459499514,45cyclictest31988-21ssh00:05:120
2459499484,7cyclictest131rcu_preempt00:25:140
2459499484,39cyclictest11958-21ssh00:30:140
2460299471,5cyclictest0-21swapper/119:50:131
2459499464,37cyclictest15221-21ssh22:25:130
24594994632,8cyclictest6230-21sh22:20:130
24594994632,6cyclictest12-21ksoftirqd/000:15:110
24594994531,7cyclictest12-21ksoftirqd/022:10:130
2459499434,32cyclictest27552-21ssh22:50:130
2459499434,32cyclictest27552-21ssh22:50:130
2461899411,25cyclictest28366irq/58-eth022:50:133
2461899411,25cyclictest28366irq/58-eth022:50:133
2461099412,18cyclictest0-21swapper/221:00:142
2459499414,33cyclictest18592-21ssh22:45:130
2461899401,22cyclictest28366irq/58-eth021:55:133
2461899401,22cyclictest28366irq/58-eth021:55:133
2460299401,4cyclictest0-21swapper/120:50:141
2461099391,21cyclictest760-21ssh22:35:112
2460299392,5cyclictest21227-21apt-get19:40:131
2460299391,29cyclictest26095-21ssh21:55:131
2460299391,29cyclictest26095-21ssh21:55:131
2460299390,3cyclictest10807-21ssh23:35:121
2459499391,28cyclictest8416-21ssh21:45:130
2461099380,29cyclictest13922-21ssh21:30:142
2461899371,20cyclictest28366irq/58-eth022:55:103
2459499371,26cyclictest0-21swapper/023:35:120
2460299361,9cyclictest27236-21kthreadcore22:30:211
2460299361,5cyclictest8413-21apt-get21:45:131
24594993620,5cyclictest12-21ksoftirqd/020:45:120
2461899358,23cyclictest0-21swapper/300:30:143
2460299351,8cyclictest4844-21grep22:35:281
2460299351,7cyclictest19748-21ssh00:34:071
2460299351,6cyclictest1815-21cut20:25:221
2459499352,25cyclictest4968-21ssh21:25:130
2461899342,17cyclictest28366irq/58-eth000:25:143
2460299341,9cyclictest29523-21tail23:25:311
2460299341,9cyclictest25492-21tr00:00:231
2460299341,9cyclictest19769-21apt23:40:121
2460299341,8cyclictest16630-21kthreadcore20:40:211
2460299341,8cyclictest14258-21kthreadcore23:00:201
2460299341,6cyclictest17514-21fschecks_time21:50:161
2460299341,5cyclictest29056-21irqrtprio21:20:191
2459499344,23cyclictest13125-21rm23:00:130
2461899333,17cyclictest28366irq/58-eth022:45:143
24610993325,4cyclictest0-21swapper/200:30:132
2460299336,3cyclictest10077-21diskmemload22:49:531
2460299331,9cyclictest23491-21tr22:10:211
2460299331,8cyclictest31155-21cat00:20:311
2460299331,8cyclictest29430-21ntp_192.168.11419:45:241
2460299331,8cyclictest29340-21munin-plugin-st19:15:011
2460299331,5cyclictest29617-21apt-get22:15:131
2460299331,10cyclictest3291-21/usr/sbin/munin00:05:251
2459499335,25cyclictest23028-21ssh00:00:150
2460299326,3cyclictest0-21swapper/123:13:271
2460299326,17cyclictest21-21ksoftirqd/121:35:121
2460299324,3cyclictest18908-21kworker/u8:021:31:451
2460299322,4cyclictest5489-21apt-get21:05:151
2460299321,9cyclictest9831-21cat23:50:301
2460299321,9cyclictest32581-21ssh23:45:291
2460299321,9cyclictest19885-21ssh23:20:231
2460299321,9cyclictest14031-21/usr/sbin/munin23:55:131
2460299321,9cyclictest11848-21kthreadcore22:40:211
2460299321,8cyclictest7857-21latency_hist20:00:011
2460299321,8cyclictest4795-21tr22:55:161
2460299321,8cyclictest31815-21/usr/sbin/munin21:40:121
2460299321,8cyclictest22791-21sed23:05:191
2460299321,8cyclictest12304-21tr19:30:181
2460299321,8cyclictest1038-21usage_Anybus19:15:291
2460299321,7cyclictest8487-21kthreadcore23:15:181
2460299321,7cyclictest26263-21ssh21:17:541
2460299321,7cyclictest13995-21tr20:05:201
2460299321,6cyclictest3081-21/usr/sbin/munin00:25:141
2460299321,5cyclictest8045-21kthreadcore21:25:211
2460299320,9cyclictest17240-21ssh22:06:141
2459499322,23cyclictest17298-21ssh21:50:140
2461899312,18cyclictest28366irq/58-eth023:40:123
2461899311,21cyclictest28366irq/58-eth020:55:123
2461899311,19cyclictest28366irq/58-eth023:30:143
2460299311,9cyclictest31756-21munin-run20:25:011
2460299311,8cyclictest18870-21fschecks_count20:45:161
2460299311,8cyclictest10077-21diskmemload22:51:281
2460299311,8cyclictest10077-21diskmemload22:51:281
2460299311,6cyclictest745-21cpuspeed21:00:141
2460299311,5cyclictest15284-21ssh22:25:151
2460299311,4cyclictest13670-21munin-run20:40:011
2460299310,9cyclictest5762-21ssh23:30:291
2460299310,8cyclictest8683-21ssh22:01:471
2460299310,6cyclictest17324-21sshd21:12:571
2460299302,8cyclictest8892-21latency_hist20:35:011
2460299301,8cyclictest28971-21ssh00:39:311
2460299301,8cyclictest28368-21cpu20:55:141
2460299301,8cyclictest22160-21/usr/sbin/munin20:15:011
2460299301,7cyclictest26230-21latency_hist00:20:011
2460299301,7cyclictest12561-21grep00:10:291
2459499303,19cyclictest30875-21rm23:10:110
24594993012,9cyclictest12-21ksoftirqd/023:05:130
2459499301,20cyclictest28471-21sh21:20:140
24618992913,9cyclictest28466irq/59-eth021:30:133
2461099292,21cyclictest10415-21ssh21:26:152
2461099292,15cyclictest22346irq/40-dwc2_hso20:55:122
2460299291,6cyclictest12576-21cat20:05:011
2460299291,5cyclictest16819-21ps19:35:181
2460299290,6cyclictest23003-21cut20:15:181
2461899283,18cyclictest0-21swapper/321:36:573
24618992811,11cyclictest0-21swapper/323:08:573
2461099283,17cyclictest22346irq/40-dwc2_hso19:55:122
2461099281,18cyclictest32111-21cut20:25:172
2460299281,9cyclictest8211-21kthreadcore19:25:201
24594992817,8cyclictest24586-21cyclictest23:40:140
2461899273,18cyclictest0-21swapper/322:06:233
2461899273,16cyclictest0-21swapper/300:02:573
2461099271,18cyclictest22346irq/40-dwc2_hso22:00:132
2459499272,19cyclictest19502-21ssh22:08:570
2459499272,19cyclictest10989-21ssh22:57:570
2461899268,12cyclictest0-21swapper/323:35:123
2461899263,14cyclictest0-21swapper/300:05:123
2459499263,18cyclictest7436-21ssh23:15:120
2459499261,19cyclictest26543-21ssh00:20:140
2461899256,9cyclictest28366irq/58-eth023:20:133
2461899254,13cyclictest28366irq/58-eth021:25:123
2461899252,14cyclictest28366irq/58-eth022:25:143
24618992512,9cyclictest28366irq/58-eth023:45:153
2461099252,9cyclictest0-21swapper/219:20:152
2461099251,8cyclictest0-21swapper/223:30:132
24610992513,5cyclictest27594-21ssh23:06:232
2459499251,18cyclictest20925-21sh00:35:160
2461899249,9cyclictest28366irq/58-eth023:50:133
2461899243,12cyclictest28366irq/58-eth000:35:153
2461899242,13cyclictest28366irq/58-eth021:45:143
2461099242,18cyclictest0-21swapper/222:50:142
2461099242,18cyclictest0-21swapper/222:50:142
2461099241,19cyclictest10077-21diskmemload22:12:572
2461099241,13cyclictest17494-21sh00:15:132
24610992411,10cyclictest0-21swapper/221:05:292
2460299241,6cyclictest24628-21grep19:10:021
2459499245,9cyclictest0-21swapper/019:35:130
2461899239,10cyclictest0-21swapper/323:10:123
2461899231,16cyclictest28366irq/58-eth021:10:123
2461099239,6cyclictest25696-21cron21:55:012
2461099232,16cyclictest0-21swapper/223:22:572
2461099232,16cyclictest0-21swapper/222:56:342
2461099232,16cyclictest0-21swapper/222:19:572
24610992312,9cyclictest28061-21ssh21:35:532
24610992312,9cyclictest13331-21ssh23:16:452
2461099231,13cyclictest9132-21irqrtprio00:10:182
2461099231,12cyclictest11501-21latency_hist19:30:022
24610992310,4cyclictest24557-21ps22:30:172
24610992310,10cyclictest0-21swapper/223:50:152
2459499232,6cyclictest0-21swapper/000:13:290
2461899228,10cyclictest28366irq/58-eth022:20:133
2461899221,13cyclictest28366irq/58-eth020:40:123
2461099222,17cyclictest21420-21grep20:10:282
2461099222,17cyclictest19772-21apt-get23:40:122
24610992211,8cyclictest23312-21switchtime21:15:282
2461099221,13cyclictest22615-21timerwakeupswit22:45:292
2461099221,12cyclictest20926-21ssh00:35:162
2461099221,12cyclictest0-21swapper/220:20:122
2459499229,3cyclictest131rcu_preempt21:10:300
2459499222,15cyclictest30825-21apt-get19:50:130
2459499221,18cyclictest7599-21ls20:30:230
2459499221,17cyclictest6785-21apt-get19:25:120
2461899213,9cyclictest28366irq/58-eth023:55:133
24618992114,3cyclictest0-21swapper/319:50:133
2461099219,9cyclictest2371-21basename22:00:012
2461099219,9cyclictest2371-21basename22:00:012
2461099219,10cyclictest2688-21cron00:25:012
2461099219,10cyclictest20880-21sshd19:37:572
2461099217,11cyclictest0-21swapper/221:45:212
2461099213,15cyclictest343-21grep23:45:312
2461099212,10cyclictest3490-21interrupts00:25:192
2461099211,10cyclictest22973-21/usr/sbin/munin00:00:152
2459499217,9cyclictest12-21ksoftirqd/020:35:130
2459499212,6cyclictest13281-21ntp_192.168.11422:40:260
2459499211,16cyclictest0-21swapper/021:15:130
2459499211,15cyclictest31868-21sh21:40:140
2459499211,14cyclictest24328-21cut22:30:150
2459499211,12cyclictest4170-21cut23:30:220
2459499210,14cyclictest28434-21cut20:55:150
2461899209,9cyclictest0-21swapper/321:40:023
2461899203,11cyclictest0-21swapper/321:15:133
2461899202,13cyclictest28366irq/58-eth020:50:143
2461899202,12cyclictest28366irq/58-eth000:20:123
2461899201,15cyclictest0-21swapper/323:25:153
24618992010,7cyclictest28366irq/58-eth022:35:123
24618992010,4cyclictest0-21swapper/319:20:153
2461099208,5cyclictest10307-21ssh21:10:032
2461099203,9cyclictest17539-21switchtime20:40:272
2461099202,9cyclictest27140-21latency_hist20:20:022
2461099202,15cyclictest1456-21kthreadcore00:05:192
2461099202,15cyclictest12972-21unixbench_singl20:35:282
2461099201,9cyclictest7345-21kthreadcore20:30:232
2461099201,9cyclictest31724-21kthreadcore23:10:182
2461099201,9cyclictest26666-21kthreadcore20:50:232
2461099201,9cyclictest13137-21kthreadcore22:05:202
24610992011,7cyclictest0-21swapper/222:26:362
2461099201,15cyclictest10478-21kthreadcore20:00:212
2461099201,11cyclictest30248-21kthreadcore21:20:212
2461099201,10cyclictest32689-21kthreadcore19:50:212
2461099201,10cyclictest32387-21irqcore21:40:192
2461099201,10cyclictest28929-21kthreadcore19:45:222
2461099201,10cyclictest22486-21grep20:45:282
2461099201,10cyclictest11978-21cut23:35:192
24610992010,3cyclictest13134-21sh23:00:142
2459499201,2cyclictest0-21swapper/023:20:250
24594992011,7cyclictest29336-21sh19:15:010
2461899193,11cyclictest0-21swapper/322:32:573
2461899193,11cyclictest0-21swapper/322:19:573
2461899193,11cyclictest0-21swapper/322:01:223
2461899192,8cyclictest0-21swapper/321:05:133
24618991911,4cyclictest0-21swapper/321:50:133
2461899191,14cyclictest0-21swapper/319:45:143
2461899191,12cyclictest0-21swapper/300:15:153
2461099191,9cyclictest9510-21kthreadcore22:20:232
2461099191,9cyclictest21545-21tail19:40:162
2461099191,8cyclictest15188-21threads19:30:272
2461099191,15cyclictest26161-21kthreadcore19:10:222
2461099191,11cyclictest25377-21apt-get23:25:142
2461099191,10cyclictest12987-21fschecks_time20:05:172
2459499195,9cyclictest0-21swapper/023:45:140
2459499193,11cyclictest0-21swapper/020:15:130
2459499192,15cyclictest29833-21sed19:45:290
2459499192,13cyclictest31015-21grep20:20:280
2459499192,13cyclictest14352-21cut21:30:180
2459499191,9cyclictest8648-21kthreadcore20:00:190
2459499191,15cyclictest4503-21grep21:00:300
24594991911,5cyclictest26847-21timerandwakeup21:35:270
2459499191,15cyclictest25405-21ssh23:25:150
2459499190,13cyclictest7725-21latency_hist20:00:010
2461899183,10cyclictest0-21swapper/322:13:583
2461899182,9cyclictest0-21swapper/319:55:143
2461899182,10cyclictest0-21swapper/320:30:153
2461899182,10cyclictest0-21swapper/320:15:143
2461899182,10cyclictest0-21swapper/319:35:143
2461899182,10cyclictest0-21swapper/319:30:133
2461899181,9cyclictest28366irq/58-eth020:05:133
2461899181,10cyclictest28366irq/58-eth022:40:133
24618991810,6cyclictest0-21swapper/321:20:043
2461099182,13cyclictest24629-21latency_hist19:10:022
2461099181,9cyclictest29789-21grep19:15:172
2459499185,9cyclictest29974-21irqcore19:15:180
2459499185,10cyclictest5169-21acpi23:50:120
2459499182,13cyclictest3013-21perl22:00:160
2459499182,13cyclictest16767-21kthreadcore20:40:210
2459499182,12cyclictest26199-21df21:55:140
2459499182,12cyclictest26199-21df21:55:140
2459499182,12cyclictest14637-21irqrtprio23:55:190
24594991810,5cyclictest0-21swapper/020:25:290
2461899179,6cyclictest0-21swapper/323:18:553
2461899179,6cyclictest0-21swapper/300:11:083
2461899179,4cyclictest0-21swapper/319:25:133
2461899178,6cyclictest28366irq/58-eth023:00:143
2461899172,10cyclictest0-21swapper/320:00:143
2461899171,11cyclictest0-21swapper/319:10:243
2459499172,10cyclictest0-21swapper/019:40:130
2459499171,13cyclictest11531-21apt-get19:30:120
2459499171,12cyclictest885-21diskstats22:35:140
2461899168,4cyclictest0-21swapper/320:10:123
2461899161,10cyclictest0-21swapper/320:20:143
2459499163,11cyclictest9376-21grep21:05:290
2459499163,11cyclictest13212-21irqcore20:05:190
2459499161,12cyclictest5376-21netstat19:20:250
2459499161,11cyclictest19492-21kthreadcore20:10:200
2461899159,3cyclictest0-21swapper/320:25:143
2459499157,5cyclictest23829-21irqcore20:50:190
2459499155,8cyclictest24524-21latency_hist19:10:010
2461899149,3cyclictest0-21swapper/321:00:273
2461899148,4cyclictest28366irq/58-eth020:45:193
2461899141,9cyclictest0-21swapper/319:40:213
2461899135,5cyclictest0-21swapper/320:35:123
2461899131,6cyclictest0-21swapper/319:15:173
2461899111,7cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional