You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-26 - 15:55

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Thu Mar 26, 2026 12:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1455529582,9sleep20-21swapper/207:09:522
1430229481,9sleep00-21swapper/007:06:340
1430028659,9sleep30-21swapper/307:06:323
1431328564,16sleep10-21swapper/107:06:431
84362630,0sleep20-21swapper/211:43:282
1466599280,28cyclictest0-21swapper/212:24:252
1466399240,19cyclictest0-21swapper/111:08:081
1465999240,24cyclictest0-21swapper/009:55:440
1466399230,23cyclictest0-21swapper/110:50:391
1465999230,0cyclictest0-21swapper/010:35:340
14669992218,3cyclictest4806-21kworker/3:212:27:233
1466599220,1cyclictest0-21swapper/209:26:312
1465999220,22cyclictest0-21swapper/010:12:270
1466999210,1cyclictest0-21swapper/311:12:583
1466599210,17cyclictest0-21swapper/211:57:172
1466599210,0cyclictest0-21swapper/209:22:542
1466999200,19cyclictest0-21swapper/310:05:133
1466599200,3cyclictest141rcu_preempt07:45:142
1466599200,20cyclictest0-21swapper/209:57:182
14663992017,2cyclictest24998-21taskset10:55:201
1466399201,18cyclictest537-21snmpd10:04:251
1466399201,18cyclictest537-21snmpd10:04:241
1466399201,18cyclictest0-21swapper/109:51:191
1466399200,1cyclictest0-21swapper/111:10:121
1466399200,1cyclictest0-21swapper/110:21:171
1465999200,2cyclictest0-21swapper/009:12:060
14665991918,1cyclictest0-21swapper/211:17:302
1466599190,17cyclictest0-21swapper/210:55:052
14659991918,1cyclictest0-21swapper/009:36:240
1465999190,1cyclictest0-21swapper/011:55:440
1465999190,1cyclictest0-21swapper/011:36:210
1465999190,18cyclictest0-21swapper/009:50:140
14669991816,2cyclictest4806-21kworker/3:212:30:233
14669991816,2cyclictest4806-21kworker/3:212:18:533
1466599183,13cyclictest0-21swapper/210:01:142
1466599183,13cyclictest0-21swapper/210:01:142
1466399180,1cyclictest0-21swapper/112:28:041
1466399180,17cyclictest0-21swapper/111:29:041
313742170,0chrt30797-21/usr/sbin/munin10:40:150
1466599171,10cyclictest0-21swapper/212:05:132
14663991715,1cyclictest0-21swapper/109:45:131
1466399171,14cyclictest13742-21ntpq10:10:201
1466999160,2cyclictest7275-21ssh11:42:353
1466999160,2cyclictest0-21swapper/309:33:433
14665991612,3cyclictest20477-21ls11:50:182
1466599161,14cyclictest11148-21sh12:04:182
1466599160,15cyclictest0-21swapper/211:14:282
1466399165,10cyclictest7445-21diskmemload11:49:501
14659991616,0cyclictest0-21swapper/012:11:360
1465999160,1cyclictest0-21swapper/009:05:120
1465999160,15cyclictest0-21swapper/011:01:570
1466999150,15cyclictest0-21swapper/310:04:503
1466999150,15cyclictest0-21swapper/310:04:503
1466999150,13cyclictest0-21swapper/312:22:273
1466599154,9cyclictest0-21swapper/207:10:122
1466599150,10cyclictest0-21swapper/212:25:122
1466399152,12cyclictest0-21swapper/109:12:071
14663991515,0cyclictest0-21swapper/109:55:341
1466399151,14cyclictest0-21swapper/112:05:291
14663991510,5cyclictest25-21ksoftirqd/111:53:411
1465999153,6cyclictest6710-21munin-run12:40:000
14659991514,1cyclictest0-21swapper/010:52:250
1466999148,2cyclictest22314-21apache_volume10:35:123
1466999146,7cyclictest28107-21cpuspeed_turbos11:55:133
14669991411,2cyclictest14210-21df08:15:163
14669991411,2cyclictest14210-21df08:15:163
14669991410,3cyclictest9183-21tr09:10:193
14669991410,3cyclictest27476-21munin-run10:00:003
1466999140,2cyclictest0-21swapper/310:21:323
1466999140,0cyclictest0-21swapper/310:42:383
14665991413,1cyclictest0-21swapper/209:35:142
14665991411,2cyclictest32-21ksoftirqd/210:20:012
14665991410,3cyclictest9686-21awk11:05:272
14665991410,3cyclictest8031-21head10:45:212
14665991410,3cyclictest5859-21grep07:55:242
14665991410,2cyclictest2424-21grep11:20:202
1466599140,14cyclictest0-21swapper/211:46:032
1466599140,14cyclictest0-21swapper/211:39:582
1466399149,5cyclictest38150irq/131-snd_hda12:21:071
1466399143,8cyclictest29483-21ssh12:15:131
1466399142,2cyclictest38150irq/131-snd_hda10:09:031
14663991413,1cyclictest0-21swapper/109:38:561
14663991411,3cyclictest38150irq/131-snd_hda12:10:151
1466399140,1cyclictest0-21swapper/111:41:491
1466399140,12cyclictest0-21swapper/109:15:491
1465999146,6cyclictest16994-21idleruntime-cro07:15:000
1465999143,3cyclictest225-21systemd-journal09:28:060
1465999141,7cyclictest362-21munin-run09:25:000
1465999141,0cyclictest0-21swapper/010:08:150
1465999140,1cyclictest141rcu_preempt11:14:340
1465999140,14cyclictest0-21swapper/011:29:160
1466999139,3cyclictest24716-21grep07:30:143
1466999138,4cyclictest146622sleep311:08:553
1466999137,3cyclictest24900-21apache_processe09:20:133
1466999135,4cyclictest0-21swapper/309:20:003
14669991310,2cyclictest7594-21wc08:00:193
14669991310,2cyclictest2261-21ls09:25:193
14669991310,2cyclictest16539-21grep07:10:253
14669991310,2cyclictest16523-21grep10:50:213
14669991310,2cyclictest14608-21grep10:30:173
14669991310,2cyclictest12913-21grep08:10:253
1466599132,10cyclictest0-21swapper/209:10:192
14665991311,1cyclictest0-21swapper/210:21:282
14665991310,2cyclictest5247-21sort09:05:002
14665991310,2cyclictest3210-21head09:45:182
14665991310,2cyclictest23434-21uname10:35:182
1466599130,5cyclictest0-21swapper/210:31:352
1466599130,2cyclictest141rcu_preempt12:31:222
1466599130,2cyclictest141rcu_preempt09:50:152
1466599130,12cyclictest0-21swapper/210:27:462
1466399139,3cyclictest15844-21cat07:10:181
1466399139,2cyclictest1319-21grep08:55:171
1466399131,7cyclictest225-21systemd-journal12:35:001
1466399131,11cyclictest14399-21grep08:15:171
1466399131,11cyclictest14399-21grep08:15:161
14663991310,2cyclictest7459-21sort10:45:171
14663991310,1cyclictest26143-21tr08:40:131
1465999139,2cyclictest3941-21grep11:40:210
1465999138,3cyclictest9475-21cut09:30:140
1465999134,3cyclictest30691-21perf12:35:000
14659991310,2cyclictest4569-21grep09:00:250
14659991310,2cyclictest19213-21grep08:25:170
14659991310,2cyclictest13-21ksoftirqd/010:31:160
1465999130,1cyclictest0-21swapper/011:21:360
1465999130,1cyclictest0-21swapper/011:16:230
1465999130,13cyclictest0-21swapper/012:03:240
1465999130,12cyclictest18733-21grep11:30:170
284492121,1sleep30-21swapper/310:57:343
1466999129,3cyclictest39-21ksoftirqd/309:39:203
1466999129,2cyclictest6931-21sed09:05:253
1466999129,2cyclictest31698-21df12:35:133
1466999129,2cyclictest31147-21grep08:50:153
1466999129,2cyclictest27061-21tune2fs09:40:183
1466999129,2cyclictest25311-21ssh11:15:143
1466999129,2cyclictest2491-21head07:50:173
1466999129,2cyclictest19364-21cron07:20:003
1466999129,2cyclictest1641-21grep11:00:263
1466999128,3cyclictest7777-21ssh10:45:183
1466999128,3cyclictest31556-21basename07:44:593
1466999123,5cyclictest225-21systemd-journal12:10:013
1466999122,9cyclictest2923-21timerwakeupswit11:20:253
1466999121,1cyclictest0-21swapper/312:10:123
1466599129,3cyclictest32-21ksoftirqd/209:30:152
1466599129,2cyclictest32420-21chrt09:43:342
1466599129,2cyclictest31329-21ls12:15:262
1466599129,2cyclictest29549-21wc08:45:212
1466599129,2cyclictest18844-21sed11:30:172
1466599129,2cyclictest15511-21grep10:50:152
1466599129,2cyclictest14445-21grep08:15:172
1466599129,2cyclictest14445-21grep08:15:172
1466599129,2cyclictest10500-21ls11:25:182
1466599120,2cyclictest0-21swapper/211:03:372
1466599120,10cyclictest0-21swapper/209:05:132
1466399129,2cyclictest9624-21grep08:05:151
1466399129,2cyclictest4279-21cut07:55:131
1466399129,2cyclictest30778-21ssh09:42:191
1466399129,2cyclictest28509-21tr08:45:141
1466399129,2cyclictest2029-21apt11:40:001
1466399129,2cyclictest1229-21wc11:00:211
1466399128,3cyclictest3735-21taskset07:53:201
1466399127,4cyclictest32420-21ssh12:35:191
1466399127,4cyclictest18062-21ssh10:32:071
1466399121,2cyclictest4053-21/usr/sbin/munin12:00:201
1466399121,11cyclictest7445-21diskmemload11:31:291
1465999129,2cyclictest9135-21tr08:05:130
1465999129,2cyclictest6587-21head12:20:150
1465999129,2cyclictest32576-21head07:45:160
1465999129,2cyclictest3196-21head09:45:180
1465999129,2cyclictest30290-21sed10:20:180
1465999129,2cyclictest28046-21cat09:40:270
1465999129,2cyclictest24401-21turbostat.cron09:20:000
1465999129,2cyclictest21232-21grep11:50:240
1465999129,2cyclictest14478-21sed12:05:260
1465999124,3cyclictest225-21systemd-journal09:00:000
1465999120,1cyclictest0-21swapper/010:16:070
1465999120,1cyclictest0-21swapper/010:00:220
1465999120,1cyclictest0-21swapper/010:00:210
1466999119,1cyclictest19044-21sort08:25:163
1466999118,2cyclictest9479-21ssh11:25:093
1466999118,2cyclictest9278-21ssh10:26:513
1466999118,2cyclictest9101-21egrep08:05:123
1466999118,2cyclictest607-21grep07:45:223
1466999118,2cyclictest4774-21ssh09:45:453
1466999118,2cyclictest4268-21wc09:00:213
1466999118,2cyclictest28967-21readlink07:40:003
1466999118,2cyclictest28540-21ssh11:35:263
1466999118,2cyclictest27960-21sh08:45:013
1466999118,2cyclictest24196-21idleruntime-cro07:30:003
1466999118,2cyclictest22482-21taskset08:30:243
1466999118,2cyclictest21490-21taskset07:21:333
1466999118,2cyclictest19202-21ssh11:50:053
1466999118,2cyclictest18505-21ssh11:30:153
1466999118,2cyclictest1557-21sed08:55:193
1466999118,2cyclictest14634-21ssh10:10:333
1466999118,2cyclictest14630-21ssh11:46:443
1466999118,2cyclictest12508-21wc09:50:253
1466999117,3cyclictest23812-21expr08:35:143
1466999117,3cyclictest21252-21ssh10:15:153
1466999110,10cyclictest5655-21sendmail_mailtr12:00:223
1466599118,2cyclictest8961-21cut08:05:112
1466599118,2cyclictest4378-21ssh12:38:122
1466599118,2cyclictest4292-21sh10:05:142
1466599118,2cyclictest31440-21grep10:40:152
1466599118,2cyclictest27933-21cat07:35:202
1466599118,2cyclictest24498-21grep07:30:122
1466599118,2cyclictest22320-21tr07:25:142
1466599118,2cyclictest21734-21grep08:30:162
1466599118,2cyclictest21434-21chrt07:20:502
1466599118,2cyclictest17972-21grep07:15:182
1466599118,2cyclictest17917-21taskset08:21:472
1466599118,2cyclictest13179-21cut10:10:152
1466599117,3cyclictest25062-21taskset08:35:262
1466599110,9cyclictest0-21swapper/212:10:562
1466599110,10cyclictest0-21swapper/208:50:122
1466399118,3cyclictest22387-21apache208:50:011
1466399118,2cyclictest8046-21cat08:00:251
1466399118,2cyclictest4032-21grep09:00:191
1466399118,2cyclictest3757-21ssh10:43:051
1466399118,2cyclictest28933-21taskset07:39:591
1466399118,2cyclictest28922-21cut11:55:181
1466399118,2cyclictest26576-21munin-run07:35:001
1466399118,2cyclictest26339-21sed11:15:201
1466399118,2cyclictest25937-21ssh09:20:191
1466399118,2cyclictest24541-21ssh10:35:331
1466399118,2cyclictest23225-21cat08:35:011
1466399118,2cyclictest20659-21ssh10:15:011
1466399118,2cyclictest18040-21head07:15:181
1466399118,2cyclictest1616-21cut09:25:151
1466399118,2cyclictest12447-21ntpq08:10:201
1466399117,1cyclictest11911-21ssh09:30:521
14663991110,1cyclictest0-21swapper/111:20:141
1466399110,10cyclictest30551-21cut07:40:181
1466399110,10cyclictest13194-21ssh10:29:531
1465999119,1cyclictest6238-21grep10:25:150
1465999118,2cyclictest8838-21ssh10:45:290
1465999118,2cyclictest7410-21cat08:00:170
1465999118,2cyclictest30179-21head07:40:160
1465999118,2cyclictest27108-21tr07:35:150
1465999118,2cyclictest23066-21chrt08:33:290
1465999118,2cyclictest20776-21ssh12:28:560
1465999118,2cyclictest20412-21head07:20:150
1465999118,2cyclictest16824-21cut08:20:170
1465999118,2cyclictest12882-21ssh11:07:320
1465999117,3cyclictest23689-21grep08:35:150
1465999116,5cyclictest262942sleep007:31:350
1465999116,2cyclictest225-21systemd-journal08:50:010
14659991110,1cyclictest0-21swapper/011:46:120
1465999110,10cyclictest27993-21ssh10:57:120
1465999110,10cyclictest1346-21ssh12:17:110
1466999107,2cyclictest4398-21grep07:55:133
1466999107,2cyclictest30255-21taskset08:49:073
1466999107,2cyclictest17167-21grep08:20:203
1466599108,1cyclictest31415-21taskset07:43:232
1466599108,1cyclictest19016-21sh08:25:152
1466599108,1cyclictest1788-21cut07:50:142
1466599108,1cyclictest132192sleep208:12:512
1466599107,2cyclictest27505-21tail08:40:252
1466599107,2cyclictest18250-21sed09:15:262
1466599107,2cyclictest1806-21wc08:55:212
1466599106,3cyclictest7213-21sort08:00:162
1466399108,1cyclictest23580-21sed07:25:241
1466399108,1cyclictest18851-21expr08:25:151
1466399108,1cyclictest17129-21sort08:20:191
1466399107,2cyclictest5707-21expr09:05:141
1466399107,2cyclictest25145-21sed08:35:261
1465999108,2cyclictest12403-21nfs4_client08:10:190
1465999108,1cyclictest31027-21expr08:50:150
1465999108,1cyclictest26290-21cat08:40:120
1465999108,1cyclictest17616-21expr07:15:160
1465999107,2cyclictest5099-21wc07:55:180
1465999107,2cyclictest1835-21cut07:50:140
1465999107,2cyclictest15295-21grep08:15:260
1465999107,2cyclictest15295-21grep08:15:250
1465999100,9cyclictest22553-21expr07:25:140
146639997,1cyclictest32159-21expr07:45:141
146639997,1cyclictest21402-21awk07:20:251
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional