You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-06 - 20:55

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Wed May 06, 2026 12:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
415929668,8sleep30-21swapper/307:05:043
618829482,8sleep00-21swapper/007:08:400
603329077,9sleep20-21swapper/207:06:402
610028467,13sleep10-21swapper/107:07:321
6382993330,1cyclictest32759-21kworker/3:112:18:423
6382993324,8cyclictest30436-21kworker/3:011:50:423
6382992825,2cyclictest22713-21kworker/3:212:39:033
636999260,26cyclictest0-21swapper/010:36:510
638299231,21cyclictest30436-21kworker/3:012:00:433
638299230,22cyclictest32759-21kworker/3:112:13:433
637999222,2cyclictest141rcu_preempt09:36:132
636999222,19cyclictest0-21swapper/010:58:260
637599211,4cyclictest0-21swapper/111:48:331
637599211,4cyclictest0-21swapper/111:48:331
637599210,21cyclictest0-21swapper/112:18:101
6382992017,1cyclictest30436-21kworker/3:010:10:073
638299200,1cyclictest0-21swapper/309:49:543
637599200,18cyclictest0-21swapper/111:16:111
636999207,7cyclictest24956-21munin-run09:25:000
636999207,12cyclictest13-21ksoftirqd/009:15:140
6382991917,1cyclictest2343-21kworker/3:310:05:063
6382991917,1cyclictest0-21swapper/310:30:133
6382991915,1cyclictest30436-21kworker/3:010:21:443
6382991817,1cyclictest30436-21kworker/3:010:18:383
638299180,16cyclictest0-21swapper/312:32:593
638299180,14cyclictest0-21swapper/311:55:123
6375991817,1cyclictest0-21swapper/111:31:121
637599181,4cyclictest225-21systemd-journal09:30:011
637599180,1cyclictest0-21swapper/109:14:321
636999186,6cyclictest21332-21munin-run10:20:010
6382991716,1cyclictest0-21swapper/309:11:553
6382991714,2cyclictest19635-21apt-config10:00:003
6382991714,2cyclictest1089-21apache210:53:513
6382991714,1cyclictest2343-21kworker/3:310:03:273
638299170,1cyclictest0-21swapper/311:01:483
637599172,2cyclictest0-21swapper/107:50:121
637599171,15cyclictest31771-21diskmemload09:16:121
638299160,9cyclictest0-21swapper/309:05:203
638299160,9cyclictest0-21swapper/309:05:193
637999164,8cyclictest0-21swapper/211:21:042
6379991614,1cyclictest0-21swapper/208:50:162
636999160,1cyclictest0-21swapper/010:29:510
636999160,1cyclictest0-21swapper/010:13:020
636999160,16cyclictest0-21swapper/012:08:520
636999160,15cyclictest0-21swapper/009:59:260
638299152,12cyclictest19273-21python11:15:243
638299150,15cyclictest0-21swapper/312:08:403
637999156,4cyclictest5116-21apt-get12:05:122
637999154,8cyclictest31771-21diskmemload12:26:492
6379991513,2cyclictest32-21ksoftirqd/210:57:422
637999150,4cyclictest0-21swapper/211:43:062
637599157,6cyclictest21534-21cat10:20:011
637599152,7cyclictest10597-21seq11:30:001
6375991514,1cyclictest0-21swapper/110:08:481
637599150,2cyclictest0-21swapper/110:52:481
636999157,7cyclictest20137-21munin-run11:55:010
636999154,3cyclictest225-21systemd-journal08:05:450
6369991513,1cyclictest0-21swapper/011:32:390
636999150,0cyclictest0-21swapper/010:33:010
638299149,5cyclictest1104-21apache212:21:283
6382991410,2cyclictest15643-21head10:35:173
638299140,1cyclictest0-21swapper/310:28:503
638299140,1cyclictest0-21swapper/309:38:283
638299140,13cyclictest4022-21head08:10:153
637999149,4cyclictest23194-21if_err_enp0s31f12:15:182
6379991410,3cyclictest31749-21cut10:25:202
6379991410,3cyclictest1750-21sed08:05:182
6379991410,3cyclictest11666-21tr07:20:152
6379991410,2cyclictest31645-21cat12:40:012
6379991410,2cyclictest0-21swapper/212:13:582
637599149,4cyclictest537-21snmpd10:44:331
637599149,4cyclictest537-21snmpd10:44:331
637599147,4cyclictest0-21swapper/110:35:151
637599146,5cyclictest24601-21cron08:55:001
637599143,1cyclictest141rcu_preempt11:24:101
6375991412,2cyclictest0-21swapper/109:55:201
6375991410,3cyclictest12942-21ssh10:15:001
637599140,1cyclictest28865-21sed09:45:251
6369991410,3cyclictest29478-21df10:05:140
6369991410,3cyclictest23300-21wc07:45:000
638299139,3cyclictest11365-21sed08:25:163
6382991310,2cyclictest7939-21cron08:19:593
6382991310,2cyclictest26475-21idleruntime-cro09:44:593
6382991310,2cyclictest21850-21grep07:40:173
6382991310,2cyclictest15461-21cut08:35:143
638299130,12cyclictest0-21swapper/311:49:453
638299130,12cyclictest0-21swapper/311:49:443
637999139,3cyclictest27199-21sh11:40:002
637999139,3cyclictest26977-21uname07:50:192
637999139,3cyclictest25623-21perf11:20:002
637999139,3cyclictest23339-21wc07:45:002
637999139,2cyclictest5570-21dpkg08:15:002
637999134,7cyclictest21801-21/usr/sbin/munin10:20:142
6379991310,2cyclictest9756-21sed07:15:182
6379991310,2cyclictest4606-21cron12:05:012
6379991310,2cyclictest28365-21cat10:05:012
6379991310,2cyclictest25566-21tune2fs08:55:172
6379991310,2cyclictest16627-21grep07:30:152
6379991310,2cyclictest15184-21grep12:30:122
637999130,6cyclictest0-21swapper/209:33:452
637999130,2cyclictest0-21swapper/212:21:202
637999130,12cyclictest16510-21sendmail09:20:002
637999130,12cyclictest0-21swapper/210:49:202
637599139,3cyclictest29515-21wc11:40:261
637599139,3cyclictest15418-21grep12:30:131
637599139,2cyclictest12895-21cat07:20:261
637599138,4cyclictest32687-21taskset08:04:201
637599137,4cyclictest6720-21ssh12:25:121
637599134,8cyclictest7007-21df_inode10:30:151
637599131,3cyclictest225-21systemd-journal10:04:591
6375991310,2cyclictest5923-21grep12:05:161
6375991310,2cyclictest3675-21grep09:30:261
6375991310,2cyclictest32183-21ssh12:20:221
6375991310,2cyclictest30330-21cut10:25:111
6375991310,2cyclictest29135-21grep12:00:151
6375991310,2cyclictest27192-21cron11:40:001
6375991310,2cyclictest23837-21tr12:35:141
6375991310,2cyclictest14875-21wc12:10:181
6375991310,2cyclictest144362sleep109:37:131
6375991310,2cyclictest14150-21ssh11:50:271
637599130,12cyclictest10294-21cut07:15:231
637599130,11cyclictest25105-21grep07:45:261
636999139,2cyclictest30840-21ls09:05:210
636999139,2cyclictest30840-21ls09:05:210
636999139,2cyclictest23355-21tr10:40:140
636999139,2cyclictest23355-21tr10:40:140
636999135,7cyclictest7957-21munin-run08:20:000
636999131,6cyclictest955-21perf09:30:000
6369991313,0cyclictest0-21swapper/011:39:450
6369991310,2cyclictest9282-21grep07:15:160
6369991310,2cyclictest26970-21cut11:20:170
6369991310,2cyclictest2647-21grep11:05:250
6369991310,2cyclictest19307-21cut09:40:160
6369991310,2cyclictest17290-21cat11:15:010
636999130,12cyclictest24214-21head07:45:170
636999130,11cyclictest0-21swapper/012:17:480
638299129,2cyclictest9968-21sort11:10:163
638299129,2cyclictest3492-21tr11:25:173
638299129,2cyclictest32464-21head10:45:173
638299129,2cyclictest29853-21chrt09:27:183
638299129,2cyclictest28025-21grep11:40:153
638299129,2cyclictest28013-21grep11:20:253
638299129,2cyclictest27495-21grep09:00:143
638299129,2cyclictest26967-21grep07:50:193
638299129,2cyclictest2573-21head09:30:163
638299129,2cyclictest1944-21grep11:05:203
638299129,2cyclictest11635-21tr07:20:153
638299128,3cyclictest867-21tr08:05:143
638299128,3cyclictest25141-21ssh10:40:283
638299128,3cyclictest25141-21ssh10:40:273
638299128,2cyclictest25598-21cron07:49:593
6382991211,1cyclictest0-21swapper/310:59:263
638299120,11cyclictest24598-21cron08:55:003
637999129,2cyclictest21390-21tr11:55:162
637999129,2cyclictest14227-21grep10:15:172
637999129,2cyclictest13467-21ssh10:53:172
637999129,2cyclictest12972-21grep11:50:162
637999129,2cyclictest11160-21ssh11:10:262
637999128,3cyclictest960-21ssh09:30:012
637999128,3cyclictest7470-21ssh10:30:182
637999128,3cyclictest14488-21head07:25:162
637999128,3cyclictest11920-21tr09:55:142
637999121,4cyclictest4294-21ls11:25:242
6379991212,0cyclictest0-21swapper/211:45:452
6379991212,0cyclictest0-21swapper/211:45:452
637999120,2cyclictest0-21swapper/211:08:102
637999120,10cyclictest0-21swapper/210:40:142
637999120,10cyclictest0-21swapper/210:40:142
637599129,2cyclictest837-21tr08:05:131
637599129,2cyclictest32522-21head10:45:181
637599129,2cyclictest30402-21tail09:05:181
637599129,2cyclictest30402-21tail09:05:171
637599129,2cyclictest28094-21rm11:59:411
637599129,2cyclictest25588-21head08:55:171
637599129,2cyclictest25508-21head11:00:171
637599129,2cyclictest22559-21cut10:20:161
637599129,2cyclictest17255-21ssh09:20:131
637599129,2cyclictest16140-21cut08:35:171
637599128,3cyclictest17132-21cron11:15:001
637599127,4cyclictest18461-21latency_hist07:35:001
636999129,3cyclictest0-21swapper/011:43:280
636999129,2cyclictest27623-21diskstats09:45:150
636999129,2cyclictest21072-21head10:00:150
636999129,2cyclictest18445-21grep11:15:160
636999129,2cyclictest13687-21tr08:30:160
636999129,2cyclictest13488-21cron07:25:000
636999128,3cyclictest6769-21cut12:25:120
636999128,2cyclictest14360-21grep07:25:150
6369991210,1cyclictest21822-21grep08:45:250
638299118,2cyclictest5502-21ssh09:50:473
638299118,2cyclictest32300-21grep08:00:253
638299118,2cyclictest22157-21ssh11:36:053
638299118,2cyclictest21678-21ssh09:22:293
638299118,2cyclictest14154-21chrt12:29:383
638299118,2cyclictest13830-21grep07:25:133
638299118,2cyclictest13121-21grep08:30:143
638299118,2cyclictest11861-21ssh09:16:243
638299117,3cyclictest29019-21sed07:55:173
638299110,10cyclictest11194-21expr11:30:143
637999118,2cyclictest9597-21cat08:20:222
637999118,2cyclictest9419-21ssh10:12:192
637999118,2cyclictest3365-21rm09:50:142
637999118,2cyclictest30283-21grep10:05:202
637999118,2cyclictest28243-21tr09:45:202
637999118,2cyclictest27867-21sort09:00:162
637999118,2cyclictest27394-21ssh11:01:012
637999118,2cyclictest2488-21ssh09:10:512
637999118,2cyclictest20309-21chrt08:45:132
637999118,2cyclictest20065-21grep09:40:212
637999118,2cyclictest19400-21tail08:40:252
637999118,2cyclictest17761-21grep09:20:152
637999118,2cyclictest17635-21wc08:40:002
637999118,2cyclictest15331-21expr10:35:152
637999118,2cyclictest11652-21tune2fs11:30:162
637599118,2cyclictest6882-21grep07:10:151
637599118,2cyclictest6882-21grep07:10:141
637599118,2cyclictest502-21ssh11:05:071
637599118,2cyclictest3841-21grep08:10:141
637599118,2cyclictest2937-21ssh09:50:091
637599118,2cyclictest29030-21expr07:55:171
637599118,2cyclictest27834-21sed09:00:151
637599118,2cyclictest21384-21wc08:45:211
637599118,2cyclictest18653-21aten2_r7power_p09:40:121
637599118,2cyclictest18299-21grep08:40:141
637599118,2cyclictest14210-21tr08:30:231
637599118,2cyclictest11318-21tail08:25:161
637599117,3cyclictest15332-21wc07:25:261
636999119,1cyclictest7323-21grep07:10:180
636999119,1cyclictest7323-21grep07:10:170
636999118,2cyclictest8958-21head08:20:170
636999118,2cyclictest7567-21ssh10:50:060
636999118,2cyclictest697-21head09:10:170
636999118,2cyclictest5306-21ssh09:31:390
636999118,2cyclictest5164-21ssh09:50:310
636999118,2cyclictest5060-21ntpq11:45:200
636999118,2cyclictest5060-21ntpq11:45:200
636999118,2cyclictest32493-21taskset08:01:510
636999118,2cyclictest3244-21grep11:25:150
636999118,2cyclictest30549-21ssh12:20:080
636999118,2cyclictest29835-21grep12:00:170
636999118,2cyclictest27001-21ssh11:00:430
636999118,2cyclictest23446-21ssh10:20:220
636999118,2cyclictest23210-21ssh11:55:530
636999118,2cyclictest19506-21expr07:35:180
636999118,2cyclictest16922-21rm12:30:240
636999118,2cyclictest16889-21ssh09:39:070
636999118,2cyclictest1294-21ssh10:45:270
636999114,5cyclictest13924-21wget12:10:130
636999114,3cyclictest23580-21apt-get12:35:130
638299108,1cyclictest25774-21if_err_enp2s008:55:183
638299108,1cyclictest18099-21cat08:40:153
638299107,2cyclictest9768-21sed08:20:253
638299107,2cyclictest7074-21expr07:10:163
638299107,2cyclictest7074-21expr07:10:153
638299107,2cyclictest20159-21grep08:45:123
638299107,2cyclictest19459-21head07:35:173
638299107,2cyclictest17096-21grep07:30:173
638299103,3cyclictest225-21systemd-journal07:20:003
637999108,1cyclictest6608-21apache_accesses07:10:132
637999108,1cyclictest6608-21apache_accesses07:10:122
637999108,1cyclictest14829-21chrt08:31:472
637999108,1cyclictest11067-21expr08:25:142
637999107,2cyclictest30603-21chrt08:00:092
637999107,2cyclictest28288-21cut07:55:142
637999107,2cyclictest25524-21chrt07:49:072
637999106,3cyclictest7454-21grep08:15:252
637999100,9cyclictest19210-21df07:35:152
637599108,1cyclictest22888-21taskset07:40:461
637599108,1cyclictest19516-21sh07:35:171
637599108,1cyclictest10095-21chrt08:21:571
637599107,2cyclictest6770-21uname08:15:181
637599107,2cyclictest16330-21tr10:55:141
636999108,1cyclictest4464-21awk08:10:200
636999108,1cyclictest27375-21grep09:00:130
636999108,1cyclictest24936-21cut08:55:130
636999108,1cyclictest19780-21taskset08:44:060
636999108,1cyclictest15369-21cut08:35:130
636999108,1cyclictest123532sleep008:25:270
636999107,2cyclictest28206-21taskset07:55:060
636999107,2cyclictest26242-21expr07:50:140
63799997,1cyclictest29786-21cpu09:05:152
63799997,1cyclictest29786-21cpu09:05:142
63699997,1cyclictest18183-21taskset07:31:360
63699996,2cyclictest23191-21grep08:50:160
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional