You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-04 - 10:22

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Thu Jun 04, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2503329280,8sleep00-21swapper/019:06:530
2505529166,9sleep20-21swapper/219:07:122
2517928871,13sleep10-21swapper/119:08:481
2497428269,9sleep30-21swapper/319:06:093
2536699280,27cyclictest21904-21kworker/3:222:53:433
25364992825,2cyclictest32-21ksoftirqd/221:40:002
2536099270,1cyclictest0-21swapper/100:12:401
2536699260,25cyclictest21904-21kworker/3:223:20:133
2536699260,25cyclictest10254-21kworker/3:121:52:133
25366992522,3cyclictest21904-21kworker/3:222:56:433
2536699250,24cyclictest21904-21kworker/3:223:52:333
25360992523,1cyclictest0-21swapper/100:20:151
2536699240,23cyclictest21904-21kworker/3:223:43:133
2536699240,23cyclictest10254-21kworker/3:121:55:433
2536699230,23cyclictest21904-21kworker/3:223:59:023
2536699230,23cyclictest21904-21kworker/3:223:34:033
2535899232,2cyclictest141rcu_preempt23:26:130
25364992221,1cyclictest0-21swapper/222:21:102
2535899224,17cyclictest477-21rs:main0
2536699210,20cyclictest0-21swapper/321:19:033
25364992120,1cyclictest0-21swapper/222:03:192
25366992019,1cyclictest0-21swapper/322:33:073
2536499200,2cyclictest141rcu_preempt21:42:192
2536499200,19cyclictest0-21swapper/221:17:552
2536099205,14cyclictest29817-21idleruntime-cro21:50:011
2536099202,2cyclictest20560-21ssh00:18:281
25360992016,3cyclictest29255-21diskmemload00:00:471
2536099200,18cyclictest0-21swapper/121:27:501
2536099200,17cyclictest0-21swapper/122:56:561
2535899207,6cyclictest22484-21awk22:05:010
25366991916,2cyclictest29255-21diskmemload23:07:283
2536499191,0cyclictest141rcu_preempt22:19:032
2536099190,19cyclictest25-21ksoftirqd/122:48:331
2536099190,19cyclictest0-21swapper/121:53:521
25366991817,1cyclictest0-21swapper/300:30:133
2536699180,18cyclictest21904-21kworker/3:222:25:033
2536499184,3cyclictest55850irq/129-enp2s0-23:40:302
25364991814,4cyclictest0-21swapper/222:47:562
2536499180,0cyclictest0-21swapper/223:25:562
2536099181,16cyclictest29255-21diskmemload00:33:551
2536099180,18cyclictest0-21swapper/123:22:031
2536099180,18cyclictest0-21swapper/122:22:231
2536099180,18cyclictest0-21swapper/122:09:231
25366991716,1cyclictest0-21swapper/323:01:023
25364991711,3cyclictest32-21ksoftirqd/223:35:172
2536499170,1cyclictest0-21swapper/222:35:112
25360991715,2cyclictest25-21ksoftirqd/123:33:221
25360991713,1cyclictest0-21swapper/100:37:371
25364991615,1cyclictest0-21swapper/221:52:402
25364991614,1cyclictest32-21ksoftirqd/222:29:572
25364991614,1cyclictest0-21swapper/200:05:132
2536499160,14cyclictest0-21swapper/223:45:432
2535899160,5cyclictest0-21swapper/022:06:350
25366991513,2cyclictest21904-21kworker/3:222:06:033
25366991512,2cyclictest0-21swapper/322:44:473
25366991511,3cyclictest5158-21perf19:50:003
25364991513,1cyclictest32-21ksoftirqd/221:25:462
2536499150,1cyclictest0-21swapper/200:36:432
2536499150,15cyclictest0-21swapper/223:07:582
2536499150,15cyclictest0-21swapper/222:50:562
2536099158,7cyclictest8816-21sh21:55:551
2536099157,7cyclictest1235-21ntpd23:50:201
25360991514,1cyclictest0-21swapper/123:26:381
2536099150,1cyclictest0-21swapper/123:48:011
2536099150,14cyclictest0-21swapper/121:42:571
2535899158,7cyclictest13-21ksoftirqd/021:53:100
25358991514,1cyclictest0-21swapper/019:30:090
2535899151,2cyclictest475-21in:imuxsock19:45:000
2535899151,2cyclictest475-21in:imuxsock19:44:590
25358991512,2cyclictest22619-21uniq21:45:150
264332140,0sleep30-21swapper/320:59:593
2536699146,4cyclictest29812-21perf21:50:013
2536499148,2cyclictest27248-21munin-plugin-st23:25:002
25364991413,1cyclictest0-21swapper/223:03:402
25364991410,3cyclictest13061-21grep20:10:212
2536499140,3cyclictest0-21swapper/221:30:122
2536499140,3cyclictest0-21swapper/221:30:112
2536499140,2cyclictest0-21swapper/200:20:062
25360991413,1cyclictest0-21swapper/123:00:121
25360991411,2cyclictest27291-21grep19:10:271
2536099140,12cyclictest28401-21chrt21:02:151
2535899142,4cyclictest0-21swapper/021:36:110
2535899140,8cyclictest2015-21awk23:10:000
2535899140,0cyclictest141rcu_preempt23:24:390
150982140,0chrt0-21swapper/221:20:512
2536699139,3cyclictest1806-21ssh00:06:583
2536699134,6cyclictest0-21swapper/322:16:213
25366991312,1cyclictest0-21swapper/321:40:523
25366991310,2cyclictest5205-21grep21:15:003
2536499139,3cyclictest32-21ksoftirqd/220:30:092
2536499133,5cyclictest29255-21diskmemload21:50:012
25364991310,2cyclictest7523-21wc00:10:192
25364991310,2cyclictest26267-21sort19:10:172
25364991310,2cyclictest14309-21apache222:57:032
2536499130,1cyclictest141rcu_preempt00:19:292
2536499130,1cyclictest0-21swapper/223:19:482
2536499130,13cyclictest0-21swapper/221:11:532
2536499130,12cyclictest24548-21wc20:50:152
2536099139,3cyclictest16435-21tr22:40:001
2536099138,4cyclictest2917-21chrt22:50:251
25360991310,2cyclictest32147-21df19:30:111
25360991310,2cyclictest12111-21awk23:15:201
2536099130,13cyclictest0-21swapper/123:09:231
2535899139,3cyclictest1433-21cut22:50:140
2535899133,2cyclictest28031-21ssh21:29:280
25358991312,1cyclictest0-21swapper/022:43:220
25358991310,2cyclictest30574-21grep21:10:140
2535899130,2cyclictest0-21swapper/023:38:030
2536699129,2cyclictest30231-21grep19:20:223
2536699129,2cyclictest21280-21head20:40:123
2536699129,2cyclictest16668-21sed00:15:273
2536699129,2cyclictest14179-21munin-plugin-st00:15:003
2536699128,3cyclictest28177-21sed21:00:223
2536699124,4cyclictest225-21systemd-journal19:55:003
2536699121,2cyclictest225-21systemd-journal22:39:593
25366991211,1cyclictest21904-21kworker/3:223:25:433
25366991211,1cyclictest0-21swapper/300:37:093
25366991210,1cyclictest23779-21grep00:20:163
2536499129,2cyclictest9016-21cut00:30:232
2536499129,2cyclictest830-21grep19:30:222
2536499129,2cyclictest8278-21sh23:51:292
2536499129,2cyclictest7046-21sed19:50:212
2536499129,2cyclictest6301-21cut21:55:142
2536499129,2cyclictest5092-21head23:30:212
2536499129,2cyclictest27931-21wc21:00:212
2536499129,2cyclictest25350-21cyclictest22:41:412
2536499128,3cyclictest9861-21ls20:00:202
2536499123,8cyclictest537-21snmpd00:26:332
25364991210,1cyclictest3654-21cat23:10:192
25364991210,1cyclictest32-21ksoftirqd/200:01:212
2536099129,2cyclictest7140-21chrt23:12:301
2536099129,2cyclictest31663-21grep21:10:231
2536099129,2cyclictest30791-21wc19:25:001
2536099129,2cyclictest22188-21grep20:40:211
2536099129,2cyclictest11995-21perf21:20:001
2536099128,3cyclictest24936-21wc20:50:211
2535899129,2cyclictest3572-21cat23:10:180
2535899129,2cyclictest29986-21grep21:30:160
2535899129,2cyclictest29986-21grep21:30:160
2535899129,2cyclictest29086-21cut23:45:130
2535899129,2cyclictest27569-21cat21:00:160
2535899129,2cyclictest25935-21cat23:05:000
2535899129,2cyclictest25813-21ssh22:25:210
2535899129,2cyclictest23375-21cat20:50:000
2535899129,2cyclictest17955-21taskset21:23:050
2535899129,2cyclictest13531-21ssh21:40:090
2535899129,2cyclictest10462-21grep22:55:150
2535899129,2cyclictest0-21swapper/023:55:120
2535899125,6cyclictest27704-21munin-run19:15:000
25358991212,0cyclictest0-21swapper/022:23:340
25358991211,1cyclictest0-21swapper/000:06:570
25358991210,2cyclictest13-21ksoftirqd/000:22:590
25358991210,1cyclictest12185-21head20:10:130
2535899120,9cyclictest0-21swapper/023:17:210
2535899120,2cyclictest0-21swapper/022:38:280
2535899120,11cyclictest2134-21grep22:30:260
2536699119,1cyclictest30478-21grep21:30:193
2536699119,1cyclictest30478-21grep21:30:193
2536699118,2cyclictest5655-21expr21:35:143
2536699118,2cyclictest4466-21sh23:50:013
2536699118,2cyclictest31744-21egrep19:30:103
2536699118,2cyclictest22086-21rm21:25:203
2536699118,2cyclictest19153-21sed20:30:223
2536699118,2cyclictest16746-21ssh23:37:303
2536699118,2cyclictest1571-21ssh22:10:583
2536699118,2cyclictest13595-21cat20:15:013
2536699117,3cyclictest23402-21ssh00:00:253
2536699117,3cyclictest14309-21apache221:20:373
25366991110,1cyclictest10254-21kworker/3:122:00:123
2536699110,10cyclictest1951-21tr19:40:003
2536499119,1cyclictest28806-21sort19:20:112
2536499118,3cyclictest32-21ksoftirqd/222:34:232
2536499118,2cyclictest5148-21munin-plugin-st19:49:592
2536499118,2cyclictest31239-21chrt19:28:512
2536499118,2cyclictest20162-21taskset20:39:312
2536499118,1cyclictest32-21ksoftirqd/222:10:132
2536499114,3cyclictest225-21systemd-journal19:40:152
2536499114,3cyclictest225-21systemd-journal19:40:152
2536499110,7cyclictest14386-21ssh23:55:172
2536099119,1cyclictest14301-21apache220:17:501
2536099118,2cyclictest9221-21wc22:15:261
2536099118,2cyclictest5281-21cut19:50:001
2536099118,2cyclictest4538-21perf19:45:001
2536099118,2cyclictest4538-21perf19:45:001
2536099118,2cyclictest414-21tr22:30:141
2536099118,2cyclictest31903-21sed22:10:161
2536099118,2cyclictest31485-21cut00:25:141
2536099118,2cyclictest30276-21wc21:30:181
2536099118,2cyclictest30276-21wc21:30:181
2536099118,2cyclictest23969-21chrt22:25:011
2536099118,2cyclictest2161-21cat19:40:011
2536099118,2cyclictest19954-21taskset21:24:401
2536099118,2cyclictest17002-21ssh22:00:451
2536099118,2cyclictest16805-21chrt22:40:081
2536099118,2cyclictest15520-21sh23:55:331
2536099118,2cyclictest13378-21chrt20:13:131
2536099118,2cyclictest12878-21tail23:35:161
2536099117,3cyclictest16195-21grep20:20:221
2536099112,5cyclictest225-21systemd-journal20:35:001
2536099110,2cyclictest0-21swapper/123:40:101
2536099110,11cyclictest0-21swapper/100:06:161
2535899119,1cyclictest27123-21grep22:45:250
2535899119,1cyclictest23776-21grep20:50:110
2535899118,3cyclictest9068-21sh00:30:250
2535899118,2cyclictest4948-21chrt19:47:370
2535899118,2cyclictest352-21cat22:10:230
2535899118,2cyclictest28336-21systemctl19:20:000
2535899118,2cyclictest17298-21rm00:35:210
2535899118,2cyclictest15708-21tr20:20:170
2535899118,2cyclictest15044-21seq00:15:150
2535899118,2cyclictest10880-21chrt20:06:360
2535899117,3cyclictest30245-21sed19:20:220
2535899112,4cyclictest225-21systemd-journal00:14:590
25358991110,1cyclictest0-21swapper/022:18:430
2536699108,2cyclictest16578-21chrt20:24:493
2536699108,1cyclictest8466-21hostname20:00:093
2536699108,1cyclictest7954-21chrt19:57:493
2536699108,1cyclictest31171-21rm00:25:013
2536699108,1cyclictest31005-21seq19:25:483
2536699108,1cyclictest255972sleep319:10:143
2536699108,1cyclictest24980-21taskset22:45:043
2536699108,1cyclictest22952-21taskset20:46:353
2536699108,1cyclictest2263-21ssh23:10:023
2536699108,1cyclictest170382sleep320:27:473
2536699108,1cyclictest13955-21taskset20:17:383
2536699108,1cyclictest10744-21ssh23:15:093
2536699104,4cyclictest16687-21fschecks_time22:20:173
2536499108,1cyclictest77992sleep219:55:482
2536499108,1cyclictest290772sleep221:07:572
2536499108,1cyclictest258882sleep220:55:422
2536499108,1cyclictest21128-21grep20:40:122
2536499107,2cyclictest24919-21grep22:05:272
2536499107,2cyclictest23065-21taskset20:48:022
2536499107,2cyclictest14130-21taskset20:19:562
2536499107,2cyclictest10777-21taskset20:05:212
2536499101,5cyclictest225-21systemd-journal19:15:322
2536499100,9cyclictest1856-21chrt19:39:262
2536099108,1cyclictest19998-21chrt20:37:211
2536099108,1cyclictest109932sleep120:08:051
2536099107,2cyclictest5013-21ssh21:35:031
2536099107,2cyclictest28883-21chrt21:05:271
2536099107,2cyclictest16877-21taskset20:25:421
2536099102,4cyclictest225-21systemd-journal19:50:171
2535899108,1cyclictest3474-21ssh23:30:030
2535899108,1cyclictest29056-21taskset21:07:400
2535899108,1cyclictest17190-21chrt20:29:440
2535899107,2cyclictest7052-21ssh23:50:310
2535899107,2cyclictest7048-21grep19:50:210
2535899107,2cyclictest5883-21ssh21:15:120
2535899107,2cyclictest21799-21gzip20:40:170
2535899107,2cyclictest21536-21ssh00:00:010
2535899107,2cyclictest10547-21sleep20:05:000
2535899105,3cyclictest21157-21expr23:40:150
2535899102,3cyclictest477-21rs:main0
2535899102,1cyclictest6234-21ssh21:55:140
2535899100,9cyclictest26158-21taskset20:59:120
253669998,1cyclictest27985-21chrt19:15:473
253669997,1cyclictest234562sleep320:50:073
253669997,1cyclictest201192sleep320:38:573
253669997,1cyclictest10776-21taskset20:05:203
253669993,5cyclictest28921-21sshd21:05:543
253669990,8cyclictest3796-21sensors19:40:193
253669990,8cyclictest3796-21sensors19:40:193
253649996,2cyclictest16187-21head20:20:222
253609997,1cyclictest310472sleep119:26:221
253609997,1cyclictest28181-21chrt19:18:161
253609997,1cyclictest229032sleep120:45:551
253609996,2cyclictest8409-21bc20:00:071
253589997,1cyclictest78122sleep019:55:580
253589997,1cyclictest13793-21taskset20:15:370
253649980,4cyclictest225-21systemd-journal20:25:392
253609986,1cyclictest25855-21chrt20:55:181
253609984,3cyclictest8127-21cron20:00:011
253589987,1cyclictest31056-21chrt19:26:290
253589986,1cyclictest16442sleep019:36:390
253589985,2cyclictest475-21in:imuxsock20:35:340
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional