You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-24 - 05:11

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Mon Aug 24, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
793421360,0sleep00-21swapper/022:39:060
2793629481,9sleep00-21swapper/019:09:260
2786529164,9sleep20-21swapper/219:08:322
2775028568,13sleep10-21swapper/119:07:021
2774228168,9sleep30-21swapper/319:06:563
33802500,0sleep20-21swapper/223:33:582
240942480,0sleep30-21swapper/322:10:043
2809199312,28cyclictest11785-21ssh23:20:000
2809199283,20cyclictest20657-21diskmemload21:10:130
2809199280,1cyclictest0-21swapper/000:28:550
2809199280,15cyclictest20657-21diskmemload22:25:480
2809199270,26cyclictest0-21swapper/021:20:150
2809699260,26cyclictest0-21swapper/100:14:151
2809199260,25cyclictest0-21swapper/021:45:130
2809199260,1cyclictest0-21swapper/000:00:110
2809199260,0cyclictest0-21swapper/023:56:330
2810299250,1cyclictest0-21swapper/223:36:012
2809199252,19cyclictest20657-21diskmemload22:05:590
2809199251,23cyclictest0-21swapper/023:00:420
2809199250,23cyclictest12029-21runrttasks23:39:130
2809699240,1cyclictest0-21swapper/122:06:191
2809199242,19cyclictest20657-21diskmemload00:06:460
2809199241,22cyclictest20657-21diskmemload21:58:040
2809199241,21cyclictest0-21swapper/022:40:170
2809199240,23cyclictest0-21swapper/022:55:250
2809199240,1cyclictest0-21swapper/023:46:470
2809199240,1cyclictest0-21swapper/023:06:260
2810299230,22cyclictest0-21swapper/220:28:132
2809199232,20cyclictest1142-21runrttasks23:29:490
2809199231,17cyclictest20657-21diskmemload22:50:480
2809199230,22cyclictest0-21swapper/022:10:210
2809199230,20cyclictest25181-21chrt22:30:090
2809199230,0cyclictest0-21swapper/023:32:550
28096992222,0cyclictest0-21swapper/123:01:001
2809199222,19cyclictest20657-21diskmemload23:41:380
2809199222,19cyclictest20657-21diskmemload23:41:380
2809199222,19cyclictest20657-21diskmemload22:45:100
2809199221,17cyclictest20657-21diskmemload22:16:010
2809199221,17cyclictest20657-21diskmemload21:29:270
2809199220,21cyclictest0-21swapper/020:40:010
2809199220,18cyclictest20657-21diskmemload00:15:030
120932220,1sleep012094-21sh22:02:230
2809699212,4cyclictest27096-21date23:10:001
2809699210,1cyclictest0-21swapper/100:05:141
2809199212,18cyclictest0-21swapper/019:15:250
2809199211,18cyclictest20657-21diskmemload21:31:140
2809199211,17cyclictest0-21swapper/020:10:130
2809199210,1cyclictest0-21swapper/000:34:500
28105992018,2cyclictest17749-21kworker/3:121:34:533
28105992017,2cyclictest17749-21kworker/3:122:17:043
2809699202,18cyclictest20657-21diskmemload23:37:481
2809199202,15cyclictest26874-21apt-get20:15:150
2809199200,19cyclictest0-21swapper/021:38:220
173132200,0sleep30-21swapper/323:41:483
173132200,0sleep30-21swapper/323:41:483
28105991916,3cyclictest17749-21kworker/3:122:48:033
28105991915,3cyclictest350-21munin-run00:29:593
2810599190,4cyclictest23467-21kworker/u8:221:58:233
28102991918,1cyclictest0-21swapper/200:17:302
28096991915,3cyclictest27613-21ssh00:26:111
2809199196,7cyclictest7322-21munin-run00:15:000
2809199195,8cyclictest12118-21apt-get23:20:020
2809199191,12cyclictest20657-21diskmemload00:35:460
2809199190,1cyclictest0-21swapper/021:52:310
2809199190,19cyclictest0-21swapper/022:21:320
2809199190,15cyclictest0-21swapper/023:12:040
28105991816,1cyclictest0-21swapper/323:30:003
28105991814,4cyclictest17749-21kworker/3:122:50:243
28102991817,1cyclictest0-21swapper/221:52:332
28102991816,2cyclictest32-21ksoftirqd/222:25:152
2809699180,18cyclictest0-21swapper/122:50:331
2809199180,1cyclictest0-21swapper/019:35:150
2809199180,13cyclictest0-21swapper/000:22:140
157142180,3chrt39-21ksoftirqd/322:24:263
2810599177,6cyclictest0-21swapper/323:17:453
28105991715,2cyclictest17749-21kworker/3:121:53:243
28105991712,4cyclictest15919-21grep00:20:003
2810299178,6cyclictest455-21dbus-daemon20:40:002
2809699171,15cyclictest20657-21diskmemload21:14:061
2809199171,16cyclictest0-21swapper/023:54:520
2809199171,15cyclictest0-21swapper/020:50:140
2809199170,1cyclictest0-21swapper/021:17:420
2809199170,16cyclictest0-21swapper/021:40:170
2810599164,12cyclictest11534-21kworker/3:021:00:333
28105991613,3cyclictest17749-21kworker/3:123:59:333
28105991611,3cyclictest55650irq/127-enp2s0-23:10:003
2810599160,16cyclictest0-21swapper/300:24:413
2810599160,16cyclictest0-21swapper/300:04:393
28102991614,1cyclictest55650irq/127-enp2s0-21:42:322
2810299160,16cyclictest0-21swapper/200:14:532
2810299160,0cyclictest0-21swapper/220:40:152
2809699162,14cyclictest13362-1kworker/u9:022:57:151
28096991610,4cyclictest25-21ksoftirqd/121:25:181
2809199161,14cyclictest928-21grep20:25:260
2809199161,14cyclictest0-21swapper/020:30:150
2809199161,13cyclictest15667-21ls21:00:000
28105991513,1cyclictest17749-21kworker/3:123:29:343
28105991511,3cyclictest17364-21sed22:05:243
28102991511,3cyclictest31058-21uname21:15:232
28102991510,1cyclictest32-21ksoftirqd/200:03:272
2809699154,3cyclictest141rcu_preempt21:36:181
28096991514,1cyclictest0-21swapper/122:42:041
28096991513,1cyclictest0-21swapper/120:35:201
2809699150,15cyclictest0-21swapper/122:13:221
2809699150,13cyclictest0-21swapper/122:23:141
2809199159,5cyclictest20615-21chrt21:09:500
2809199159,5cyclictest20469-21head20:00:170
2809199150,8cyclictest18074-21dpkg21:05:000
2809199150,1cyclictest0-21swapper/020:20:220
2810599146,5cyclictest0-21swapper/321:40:133
2810599143,1cyclictest20657-21diskmemload23:50:153
28105991411,3cyclictest17749-21kworker/3:123:39:133
28105991411,3cyclictest17749-21kworker/3:122:37:133
28105991411,2cyclictest6345-21wc20:40:003
28105991411,2cyclictest31839-21head19:15:193
28105991411,2cyclictest10073-21grep00:35:193
2810299143,2cyclictest141rcu_preempt23:49:302
28102991413,1cyclictest0-21swapper/219:25:142
28102991413,1cyclictest0-21swapper/200:26:232
2810299140,1cyclictest0-21swapper/223:19:172
2810299140,12cyclictest0-21swapper/221:33:472
2809699149,3cyclictest25938-21wc20:10:251
28096991413,1cyclictest0-21swapper/123:53:401
2809699140,14cyclictest0-21swapper/121:31:311
2809699140,13cyclictest1545-21cpuspeed_turbos22:35:141
2809199148,5cyclictest30254-21chrt19:12:270
2809199148,5cyclictest19205-21taskset19:56:550
2809199148,5cyclictest13817-21grep19:45:220
2809199148,5cyclictest10822-21taskset20:49:060
2809199147,3cyclictest273-21systemd-udevd19:44:030
2809199141,12cyclictest22610-21df_abs20:05:150
28091991410,3cyclictest4627-21wc19:25:270
2809199140,13cyclictest0-21swapper/020:42:140
2809199140,13cyclictest0-21swapper/019:30:200
294762130,1sleep30-21swapper/323:49:403
2810599139,3cyclictest31499-21grep21:15:283
2810599139,3cyclictest29432-21ssh23:10:253
2810599139,3cyclictest21656-21ssh21:29:543
2810599139,3cyclictest20872-21tr20:00:193
2810599139,2cyclictest4117-21wc22:55:273
28105991310,2cyclictest29927-21ssh00:08:573
28105991310,2cyclictest25724-21tr20:10:203
28105991310,2cyclictest11053-21wc20:50:003
2810299138,2cyclictest21805-21diskstats21:10:152
2810299131,4cyclictest225-21systemd-journal22:00:002
28102991310,2cyclictest7558-21cron19:34:592
28102991310,2cyclictest30523-21ls20:20:222
28102991310,2cyclictest20781-21cat23:05:262
28102991310,2cyclictest20637-21sh21:10:002
28102991310,2cyclictest2029-21grep22:35:162
2810299130,2cyclictest225-21systemd-journal22:32:122
2810299130,1cyclictest141rcu_preempt22:17:212
2810299130,13cyclictest0-21swapper/222:05:282
2809699139,2cyclictest2376-21grep20:30:161
28096991312,1cyclictest0-21swapper/122:30:161
28096991310,2cyclictest12838-21grep20:50:271
2809699130,12cyclictest23674-21tr21:50:161
2809199130,0cyclictest0-21swapper/019:50:140
2810599129,2cyclictest957-21cut00:30:143
2810599129,2cyclictest29974-21cut20:20:173
2810599128,3cyclictest13376-21head19:45:183
2810599127,3cyclictest22979-21cat21:50:003
2810599123,9cyclictest56550irq/124-enp0s3121:12:473
28105991210,2cyclictest17749-21kworker/3:121:23:433
2810599120,11cyclictest18994-21cat21:05:153
2810299129,2cyclictest31458-21grep19:15:162
2810299129,2cyclictest31286-21head23:50:172
2810299129,2cyclictest28478-21grep23:10:162
2810299129,2cyclictest26593-21cron20:15:012
2810299129,2cyclictest20903-21aten2_r7power_c23:25:112
2810299129,2cyclictest16168-21grep21:00:132
2810299129,2cyclictest15395-21ssh21:25:222
2810299129,2cyclictest15199-21tr19:50:142
2810299129,2cyclictest11007-21ls22:40:222
2810299129,2cyclictest10098-21cut00:35:192
2810299129,1cyclictest0-21swapper/221:45:462
2810299128,3cyclictest27565-21ssh22:50:182
2810299123,8cyclictest11626-21apt-get23:20:022
2810299120,1cyclictest0-21swapper/222:04:142
2810299120,12cyclictest0-21swapper/222:12:482
2809699129,2cyclictest8365-21grep22:00:161
2809699129,2cyclictest4756-21grep19:25:261
2809699129,2cyclictest38150irq/131-snd_hda20:25:211
2809699129,2cyclictest31843-21uname19:15:191
2809699129,2cyclictest26139-21chrt23:47:031
2809699129,2cyclictest20503-21ssh23:05:231
2809699129,2cyclictest19944-21grep22:45:271
2809699129,2cyclictest17635-21tr19:55:161
2809699129,2cyclictest14942-21wc23:40:201
2809699129,2cyclictest14942-21wc23:40:191
2809699129,2cyclictest10036-21ssh00:15:271
2809699121,2cyclictest225-21systemd-journal22:20:001
28096991210,1cyclictest81492sleep119:35:161
2809699120,2cyclictest0-21swapper/100:35:521
2809199121,7cyclictest225-21systemd-journal19:24:140
2810599118,2cyclictest9431-21ssh22:00:253
2810599118,2cyclictest8406-21sh19:35:163
2810599118,2cyclictest6932-21grep19:30:263
2810599118,2cyclictest31343-21sed21:35:163
2810599118,2cyclictest26506-21grep22:30:193
2810599118,2cyclictest18483-21tr19:55:223
2810599118,2cyclictest16915-21ssh22:25:103
2810599118,2cyclictest16232-21ssh22:44:053
2810599118,2cyclictest16028-21ssh23:21:443
2810599118,2cyclictest14323-21head20:55:183
2810599118,2cyclictest11450-21tr20:50:153
2810599118,2cyclictest10581-21ssh23:00:093
2810599117,3cyclictest8526-21cron20:45:013
2810599117,3cyclictest22253-21grep20:05:143
2810599113,5cyclictest455-21dbus-daemon00:10:123
2810299118,2cyclictest5404-21ssh21:20:052
2810299118,2cyclictest5026-21sh00:32:062
2810299118,2cyclictest32140-21grep21:35:222
2810299118,2cyclictest29510-21rm00:08:422
2810299118,2cyclictest2817-21idleruntime-cro19:25:002
2810299118,2cyclictest2741-21expr22:55:162
2810299118,2cyclictest21594-21ssh22:46:432
2810299118,2cyclictest18661-21ssh00:20:382
2810299118,2cyclictest13686-21cat19:45:212
2810299118,2cyclictest13390-21ssh23:40:072
2810299118,2cyclictest13390-21ssh23:40:062
2810299118,2cyclictest12937-21rm23:00:392
2810299118,2cyclictest12296-21ssh23:58:312
2810299118,2cyclictest10744-21rm22:20:322
2809699119,1cyclictest974-21cat21:55:271
2809699118,2cyclictest965-21grep19:20:141
2809699118,2cyclictest9500-21chrt21:41:051
2809699118,2cyclictest8842-21taskset21:21:241
2809699118,2cyclictest8774-21ssh23:17:471
2809699118,2cyclictest5687-21grep19:30:151
2809699118,2cyclictest30528-21cat23:30:231
2809699118,2cyclictest29092-21grep19:10:161
2809699118,2cyclictest2860-21sed00:30:261
2809699118,2cyclictest21568-21uniq23:25:141
2809699118,2cyclictest20479-21grep20:00:161
2809699118,2cyclictest18840-21grep22:25:241
2809699118,2cyclictest15868-21head21:45:171
2809699118,2cyclictest15205-21tr19:50:141
2809699118,2cyclictest13073-21grep23:20:171
2809699118,2cyclictest11884-21grep19:40:271
2809699117,3cyclictest22238-21ssh00:04:381
2809699116,4cyclictest30317-21ssh21:15:171
2809699116,4cyclictest29749-21expr20:20:141
2809699116,4cyclictest14405-21sshd19:46:141
2809699116,2cyclictest18008-21ssh00:20:241
2809699115,1cyclictest23218-21grep20:05:191
2809699113,4cyclictest225-21systemd-journal20:50:001
2809699110,10cyclictest8634-21sh23:55:371
2809699110,10cyclictest10562-21kworker/1:221:05:011
2810599108,1cyclictest32507-21hostname20:25:173
2810599108,1cyclictest28347-21cat20:15:273
2810599107,2cyclictest983-21grep19:20:153
2810599107,2cyclictest4534-21sort19:25:263
2810599107,2cyclictest3249-21cat20:30:263
2810599107,2cyclictest28263-21taskset19:10:103
2810299108,1cyclictest28781-21expr19:10:142
2810299108,1cyclictest2572-21interrupts20:30:192
2810299108,1cyclictest22246-21cut20:05:142
2810299108,1cyclictest215862sleep220:01:442
2810299108,1cyclictest18362-21cat19:55:202
2810299108,1cyclictest12254-21taskset19:44:062
2810299107,2cyclictest8620-21head19:35:172
2810299107,2cyclictest14577-21cat20:55:202
2810299107,2cyclictest11549-21expr20:50:162
2810299102,7cyclictest27661-21hddtemp_smartct20:15:192
2809699108,1cyclictest6730-21cat20:40:141
2809699107,2cyclictest13838-21cut20:55:151
2809699100,1cyclictest0-21swapper/121:03:091
281059997,1cyclictest11337-21cut19:40:213
281059997,1cyclictest0-21swapper/319:50:133
281029996,2cyclictest8904-21grep20:45:142
280969996,2cyclictest27323-21expr20:15:161
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional