You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-22 - 21:08

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Tue Sep 22, 2026 12:43:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1263329452,38sleep00-21swapper/007:09:160
1267629369,19sleep10-21swapper/107:09:491
1255728471,9sleep30-21swapper/307:08:183
1056128471,9sleep20-21swapper/207:05:082
1277999261,2cyclictest141rcu_preempt12:15:282
1277999261,2cyclictest141rcu_preempt12:15:272
1277299240,2cyclictest0-21swapper/011:01:060
1277299230,20cyclictest0-21swapper/010:36:030
12779992220,2cyclictest32-21ksoftirqd/209:10:142
1277999220,22cyclictest0-21swapper/211:27:502
12776992219,2cyclictest0-21swapper/109:47:171
1277699220,1cyclictest0-21swapper/111:20:261
1278299200,20cyclictest0-21swapper/312:16:463
1278299200,20cyclictest0-21swapper/312:16:463
1278299200,1cyclictest0-21swapper/311:44:453
1278299200,1cyclictest0-21swapper/309:49:083
12776992018,1cyclictest0-21swapper/109:35:211
12776992017,1cyclictest5733-21diskmemload10:56:421
1277699200,18cyclictest0-21swapper/111:30:141
1277299205,7cyclictest17526-21munin-run07:20:000
12779991918,1cyclictest0-21swapper/208:15:132
1277699193,15cyclictest5733-21diskmemload11:40:451
1277699190,1cyclictest0-21swapper/112:22:191
12772991918,1cyclictest0-21swapper/010:16:220
1277299190,1cyclictest0-21swapper/012:33:240
137402180,1chrt0-21swapper/009:53:240
1278299180,1cyclictest0-21swapper/312:28:303
1278299180,0cyclictest0-21swapper/311:17:503
1277999180,18cyclictest0-21swapper/209:37:462
12776991813,1cyclictest25-21ksoftirqd/109:34:461
1277299180,17cyclictest0-21swapper/012:02:290
1278299170,1cyclictest0-21swapper/309:52:063
1277999176,11cyclictest30973-21kworker/2:010:40:132
12779991715,1cyclictest0-21swapper/209:35:002
1277699176,3cyclictest537-21snmpd10:04:161
1277699173,11cyclictest5733-21diskmemload10:40:111
1277699170,17cyclictest0-21swapper/109:52:571
1277299179,7cyclictest2354-21sh11:23:530
12782991615,1cyclictest0-21swapper/310:58:203
12782991614,0cyclictest0-21swapper/309:29:323
1278299160,16cyclictest0-21swapper/311:20:523
12779991614,1cyclictest32-21ksoftirqd/209:44:562
12779991611,4cyclictest22542-21apt-get07:30:132
1277999160,16cyclictest0-21swapper/210:57:492
12776991610,1cyclictest25-21ksoftirqd/110:34:191
12782991510,3cyclictest26458-21ls07:35:263
1277999150,1cyclictest0-21swapper/211:38:282
1277699150,0cyclictest141rcu_preempt10:17:001
1277299153,6cyclictest24044-21munin-run10:00:010
1277299153,10cyclictest0-21swapper/012:20:110
12772991512,2cyclictest0-21swapper/011:11:560
1277299150,14cyclictest0-21swapper/011:31:490
1277299150,12cyclictest0-21swapper/012:10:060
12782991411,2cyclictest5733-21diskmemload11:02:393
12782991410,3cyclictest992-21cat09:00:003
12782991410,3cyclictest17494-21cron07:20:003
12782991410,2cyclictest19659-21grep10:35:213
1278299140,1cyclictest0-21swapper/312:22:193
1278299140,14cyclictest0-21swapper/310:03:543
1278299140,0cyclictest0-21swapper/312:09:513
1277999147,6cyclictest8804-21apt-get10:10:162
12779991412,1cyclictest32-21ksoftirqd/210:06:322
12779991412,1cyclictest32-21ksoftirqd/209:29:272
12779991411,2cyclictest17023-21sed12:30:202
12779991410,3cyclictest29604-21grep08:50:182
12779991410,2cyclictest26123-21grep10:20:172
1277699148,5cyclictest20255-21ssh11:53:591
1277699147,6cyclictest25-21ksoftirqd/110:39:201
12776991413,1cyclictest1948-21kworker/u8:211:29:231
1277699141,2cyclictest141rcu_preempt10:48:531
12776991412,0cyclictest0-21swapper/110:24:561
12776991411,2cyclictest27023-21dump-pmu-power07:40:001
12776991410,3cyclictest2325-21grep07:55:151
12776991410,3cyclictest16838-21wc07:15:271
12776991410,3cyclictest10582-21ssh11:47:551
12776991410,3cyclictest10376-21head08:10:181
1277699140,13cyclictest0-21swapper/110:25:011
1277699140,12cyclictest0-21swapper/110:13:501
1277299149,3cyclictest4006-21wc10:45:250
1277299146,7cyclictest23224-21munin-run09:40:000
12772991410,2cyclictest15049-21expr09:15:160
1278299138,3cyclictest23860-21sh08:40:003
1278299134,4cyclictest0-21swapper/309:40:003
12782991312,1cyclictest0-21swapper/310:20:013
12782991310,2cyclictest16603-21sed12:10:273
12782991310,2cyclictest16474-21cut09:55:163
12782991310,2cyclictest15528-21cut09:15:183
1278299130,3cyclictest23269-21cpuspeed_turbos09:20:163
1278299130,2cyclictest0-21swapper/310:31:503
1278299130,1cyclictest0-21swapper/310:47:513
12779991311,1cyclictest0-21swapper/211:40:152
12779991310,2cyclictest25669-21sed07:35:172
12779991310,2cyclictest15625-21grep07:15:162
12779991310,2cyclictest14731-21kworker/u8:210:25:492
1277999130,1cyclictest0-21swapper/211:14:062
1277999130,12cyclictest12178-21wc10:50:262
1277699139,3cyclictest26269-21perf08:45:001
1277699139,3cyclictest0-21swapper/111:15:001
1277699139,2cyclictest28559-21wc07:40:191
12776991311,1cyclictest0-21swapper/107:45:151
12776991310,2cyclictest22483-21chrt12:14:581
12776991310,2cyclictest14122-21cron08:20:001
1277699130,0cyclictest0-21swapper/109:15:531
1277299139,3cyclictest28827-21sendmail-msp11:40:000
1277299139,3cyclictest28627-21ls08:50:000
1277299139,3cyclictest23541-21ssh11:55:230
1277299139,3cyclictest1731-21cron07:55:000
1277299137,5cyclictest18003-21munin-node10:22:230
12772991310,2cyclictest7710-21grep12:05:220
12772991310,2cyclictest4877-21grep08:00:150
12772991310,2cyclictest15163-21sh12:30:000
12772991310,2cyclictest14143-21ls08:20:000
1277299130,11cyclictest0-21swapper/011:06:500
1278299129,2cyclictest9357-21wc10:30:003
1278299129,2cyclictest7082-21sed11:45:273
1278299129,2cyclictest29861-21cron12:00:003
1278299129,2cyclictest280342sleep311:39:263
1278299129,2cyclictest27704-21ssh10:40:193
1278299129,2cyclictest2712-21grep07:55:163
1278299129,2cyclictest18650-21sed08:25:263
1278299129,2cyclictest1749-21apt-get07:55:013
1278299129,2cyclictest14121-21cron08:20:003
1278299128,3cyclictest25949-21sed12:35:253
1278299125,5cyclictest1062-21df_inode10:05:163
1278299124,3cyclictest27308-21apt-get07:40:113
1278299120,11cyclictest0-21swapper/312:03:133
1277999129,2cyclictest32-21ksoftirqd/209:15:152
1277999129,2cyclictest31242-21head12:00:182
1277999129,2cyclictest30459-21head07:45:182
1277999129,2cyclictest28701-21ls11:00:212
1277999129,2cyclictest24586-21ssh10:00:132
1277999129,2cyclictest22419-21cat07:30:002
1277999129,2cyclictest20000-21grep10:35:252
1277999129,2cyclictest19750-21chrt11:15:002
1277999129,2cyclictest15843-21sed12:10:202
1277999128,3cyclictest25073-21cat12:35:182
1277999128,3cyclictest10625-21grep10:30:212
1277999126,2cyclictest225-21systemd-journal07:24:592
1277999124,3cyclictest22087-21apt-get11:55:132
1277999123,2cyclictest141rcu_preempt09:20:572
1277999122,3cyclictest141rcu_preempt12:24:302
1277999121,5cyclictest21571-21dump-pmu-power11:55:002
12779991211,1cyclictest0-21swapper/211:20:422
12779991210,1cyclictest4116-21sed11:05:182
12779991210,1cyclictest12381-21ssh12:27:492
1277699129,2cyclictest31392-21wc09:25:141
1277699129,2cyclictest25964-21head12:35:251
1277699129,2cyclictest25-21ksoftirqd/112:18:301
1277699129,2cyclictest25-21ksoftirqd/112:18:301
1277699129,2cyclictest15169-21ls12:30:001
1277699129,2cyclictest13572-21users11:10:251
1277699128,3cyclictest542-21chrt07:50:191
1277699127,2cyclictest28921-21date11:40:001
1277699122,9cyclictest23696-21ssh09:40:121
1277699122,7cyclictest38150irq/131-snd_hda08:30:141
12776991210,1cyclictest0-21swapper/112:34:361
1277699120,12cyclictest0-21swapper/109:57:481
1277699120,12cyclictest0-21swapper/107:25:161
1277299129,2cyclictest9073-21tr10:10:170
1277299129,2cyclictest3848-21grep09:05:140
1277299129,2cyclictest3761-21wc07:55:270
1277299129,2cyclictest31782-21sh07:50:000
1277299129,2cyclictest31403-21cut08:55:150
1277299129,2cyclictest29591-21cut11:40:140
1277299129,2cyclictest2892-21sed09:00:280
1277299129,2cyclictest27554-21grep07:40:130
1277299129,2cyclictest27173-21head10:40:140
1277299129,2cyclictest24916-21cut08:40:170
1277299129,2cyclictest20139-21grep08:30:170
1277299129,2cyclictest17752-21sed08:25:170
1277299129,2cyclictest15678-21grep11:50:290
1277299127,3cyclictest7776-21ssh09:30:180
1277299125,6cyclictest9103-21apt-get08:10:000
1277299124,4cyclictest225-21systemd-journal12:40:000
12772991211,1cyclictest0-21swapper/010:53:510
1278299118,2cyclictest8284-21ssh09:10:293
1278299118,2cyclictest5382-21ssh11:05:333
1278299118,2cyclictest5085-21head11:25:173
1278299118,2cyclictest2925-21sed09:00:273
1278299118,2cyclictest26769-21expr08:45:153
1278299118,2cyclictest23698-21ssh09:40:113
1278299118,2cyclictest23659-21tr07:30:193
1278299118,2cyclictest19651-21cut08:30:163
1278299118,2cyclictest18369-21diskstats07:20:163
1278299118,2cyclictest17279-21ssh10:15:153
1278299118,2cyclictest16407-21uniq12:30:153
1278299118,2cyclictest15453-21ssh11:50:283
1278299118,2cyclictest13171-21grep11:30:143
1278299118,2cyclictest12367-21grep11:10:163
1278299118,2cyclictest11474-21ssh10:10:593
1278299118,2cyclictest10382-21tr08:10:193
1278299115,2cyclictest225-21systemd-journal07:29:593
12782991111,0cyclictest0-21swapper/310:50:173
1278299110,10cyclictest7935-21cut09:30:203
1277999118,2cyclictest971-21ssh09:45:232
1277999118,2cyclictest7883-21ssh09:50:112
1277999118,2cyclictest6533-21ssh10:47:132
1277999118,2cyclictest6194-21head11:45:182
1277999118,2cyclictest32660-21python08:55:252
1277999118,2cyclictest27539-21cut07:40:122
1277999118,2cyclictest24269-21basename08:40:142
1277999118,2cyclictest21916-21ssh09:58:192
1277999118,2cyclictest20704-21sendmail_mailst08:30:242
1277999118,2cyclictest20641-21wc11:35:002
1277999118,2cyclictest18382-21ntpq10:15:212
1277999118,2cyclictest14781-21expr08:20:132
1277999118,2cyclictest11001-21ssh12:07:312
1277999114,3cyclictest225-21systemd-journal08:36:372
1277999110,3cyclictest225-21systemd-journal09:05:002
1277699118,2cyclictest30514-21hostname12:00:151
1277699118,2cyclictest29107-21cat11:00:261
1277699118,2cyclictest22685-21sed08:35:191
1277699118,2cyclictest22324-21tr11:55:141
1277699118,2cyclictest12153-21head10:50:261
1277699118,2cyclictest10873-21ssh09:12:281
1277699118,2cyclictest1038-21grep10:05:161
1277699117,3cyclictest9114-21awk08:10:011
1277699117,3cyclictest521-21grep08:55:271
1277699116,4cyclictest24714-21cat09:20:271
1277699114,2cyclictest0-21swapper/111:06:051
1277699112,3cyclictest14241-21sh12:10:001
1277299118,2cyclictest9637-21ssh10:30:090
1277299118,2cyclictest8443-21ssh09:10:360
1277299118,2cyclictest7451-21ssh11:26:210
1277299118,2cyclictest6754-21ssh10:08:510
1277299118,2cyclictest5565-21ssh11:45:160
1277299118,2cyclictest4941-21ssh10:26:370
1277299118,2cyclictest28891-21apache_processe08:50:130
1277299118,2cyclictest27312-21ssh10:01:020
1277299118,2cyclictest26592-21ssh09:21:450
1277299118,2cyclictest25786-21if_enp2s007:35:180
1277299118,2cyclictest23874-21munin-plugin-st08:40:000
1277299118,2cyclictest22423-21ssh11:15:420
1277299118,2cyclictest20113-21ssh10:55:180
1277299118,2cyclictest1776-21ssh09:45:320
1278299108,1cyclictest57032sleep309:09:553
1278299108,1cyclictest31179-21date08:55:003
1278299108,1cyclictest24162-21cut08:40:143
1278299108,1cyclictest15458-21cat08:20:183
1278299107,2cyclictest5033-21expr08:00:163
1278299107,2cyclictest31317-21sed07:45:273
1278299107,2cyclictest12977-21taskset07:10:123
1278299100,9cyclictest8143-21taskset08:05:223
1277999108,2cyclictest16829-21apt-config08:25:002
1277999108,1cyclictest32392-21expr07:50:152
1277999108,1cyclictest27907-21taskset08:45:242
1277999108,1cyclictest2477-21expr07:55:162
1277999108,1cyclictest13004-21apache_accesses07:10:122
1277999107,2cyclictest9765-21grep08:10:162
1277999107,2cyclictest4622-21aten2_r7power_c08:00:142
1277999107,2cyclictest3920-21cat09:05:142
1277999100,9cyclictest88402sleep208:06:512
1277699108,1cyclictest30505-21sed08:50:271
1277699108,1cyclictest26578-21cut08:45:151
1277699108,1cyclictest22829-21expr07:30:141
1277699108,1cyclictest188112sleep108:26:171
1277699107,2cyclictest14365-21sendmail_mailtr07:10:221
1277699102,4cyclictest225-21systemd-journal07:23:161
1277699102,3cyclictest225-21systemd-journal08:25:001
1277299108,1cyclictest30955-21sh09:25:010
1277299108,1cyclictest246202sleep007:34:470
1277299108,1cyclictest23642-21ssh09:40:090
1277299108,1cyclictest18676-21cat07:20:180
1277299107,2cyclictest23459-21expr12:15:140
1277299107,2cyclictest23459-21expr12:15:140
1277299107,2cyclictest15742-21awk08:20:220
1277299107,2cyclictest13239-21tr07:10:140
1277299107,2cyclictest11511-21chrt08:12:100
127769997,1cyclictest4983-21ls08:00:161
127769997,1cyclictest1714-21expr09:00:161
127769996,3cyclictest25-21ksoftirqd/109:10:011
127729997,1cyclictest21536-21tr07:25:230
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional