You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-11 - 14:35

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Fri Sep 11, 2026 00:45:38)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
604996991317,3cyclictest131rcu_preempt22:40:021
604992999027,16cyclictest613868-21ssh21:15:250
604992999018,67cyclictest16850irq/134-800000023:08:430
604992998428,52cyclictest688998-21ssh23:58:130
604992998425,14cyclictest611596-21ssh21:10:250
604992998425,14cyclictest611596-21ssh21:10:250
604992998228,10cyclictest623661-21ssh21:36:570
60499299821,33cyclictest632621-21ssh21:56:170
604992998128,12cyclictest650080-21ssh22:34:110
604992998128,12cyclictest650080-21ssh22:34:110
604992998116,60cyclictest16850irq/134-800000000:12:040
604992998011,6cyclictest16850irq/134-800000022:52:360
604992998011,6cyclictest16850irq/134-800000022:52:360
604992997936,40cyclictest16850irq/134-800000021:48:010
604992997927,2cyclictest0-21swapper/023:35:460
604992997920,12cyclictest673972-21ssh23:25:240
604992997920,12cyclictest673972-21ssh23:25:240
60499299791,28cyclictest0-21swapper/000:23:080
604992997726,8cyclictest696925-21ssh00:15:120
604992997724,13cyclictest692736-21ssh00:06:030
604992997723,15cyclictest626275-21ssh21:42:320
604992997618,48cyclictest16850irq/134-800000023:17:080
604992997618,48cyclictest16850irq/134-800000023:17:080
604992997617,47cyclictest16850irq/134-800000022:01:210
604992997617,47cyclictest16850irq/134-800000022:01:200
60499299761,4cyclictest48-21kcompactd023:40:250
604992997518,51cyclictest16850irq/134-800000022:58:070
604992997418,13cyclictest676164-21ssh23:30:180
604992997417,47cyclictest16850irq/134-800000021:52:550
60499299733,65cyclictest606076-21ls19:30:010
60499299731,8cyclictest636869-21ssh22:05:180
604992997318,11cyclictest707952-21ssh00:39:420
60499299731,68cyclictest617159-21ssh21:22:500
60499299731,2cyclictest0-21swapper/000:26:110
60499299731,29cyclictest0-21swapper/022:22:320
60499299731,29cyclictest0-21swapper/022:22:310
604992997236,32cyclictest16850irq/134-800000022:26:390
604992997220,12cyclictest705433-21ssh00:34:140
604992997216,46cyclictest16850irq/134-800000022:14:350
604992997216,12cyclictest663408-21ssh23:02:440
604992997216,12cyclictest663408-21ssh23:02:440
604992997130,37cyclictest16950irq/224-800000020:06:230
604992997125,11cyclictest641876-21ssh22:16:050
604992997122,13cyclictest687101-21ssh23:54:150
604992997021,14cyclictest618366-21ssh21:25:050
604992997015,9cyclictest671826-21ssh23:20:520
60500399692,16cyclictest629296-21sshd21:49:243
604992996918,46cyclictest16850irq/134-800000022:40:500
604992996918,46cyclictest16850irq/134-800000022:40:500
60500399683,3cyclictest131rcu_preempt22:15:033
604992996820,16cyclictest684973-21ssh23:49:430
604992996820,12cyclictest667437-21ssh23:11:320
60499299681,63cyclictest611375-21cut21:10:030
604996996724,5cyclictest634836-21sh22:00:571
604996996724,5cyclictest634836-21sh22:00:561
60500399668,5cyclictest131rcu_preempt20:00:003
60500399668,5cyclictest131rcu_preempt20:00:003
605003996618,43cyclictest275-21dbus-daemon21:30:003
605003996538,22cyclictest275-21dbus-daemon00:00:013
60500399653,57cyclictest643410-21cron22:19:593
604996996526,7cyclictest23-21ksoftirqd/121:36:541
604996996526,35cyclictest609797-21latency_hist20:40:021
604996996525,5cyclictest640561-21sh22:13:351
604992996515,45cyclictest16850irq/134-800000022:35:430
60500399642,20cyclictest610362-21cron20:54:593
604996996422,10cyclictest689818-21sed00:00:021
604996996419,9cyclictest686081-1kworker/u9:000:11:121
60499299642,5cyclictest1-21systemd22:50:010
605003996324,34cyclictest673598-21cat23:25:013
605003996322,36cyclictest696612-21latency_hist00:15:023
60500399632,16cyclictest701196-21munin-run00:25:013
60500399632,14cyclictest698869-21munin-run00:20:023
60500399632,14cyclictest660466-21ssh22:56:173
60500399632,10cyclictest608822-21cron20:25:013
605003996311,3cyclictest0-21swapper/319:45:423
605003996311,3cyclictest0-21swapper/319:45:423
60500399631,12cyclictest693272-21sshd00:07:233
604996996328,31cyclictest656581-21sh22:48:001
604996996322,4cyclictest611163-21diskmemload21:58:421
60499699632,18cyclictest610388-21unin-run20:55:001
60499299639,6cyclictest16850irq/134-800000000:02:350
604992996317,13cyclictest275-21dbus-daemon20:00:020
604992996317,13cyclictest275-21dbus-daemon20:00:010
605003996220,38cyclictest703539-21latency_hist00:30:013
60500399621,9cyclictest658862-21ssh22:52:573
60500399621,9cyclictest658862-21ssh22:52:563
60500399621,16cyclictest48-21kcompactd022:45:153
60500399621,15cyclictest705695-21ssh00:34:573
60500399621,14cyclictest680228-21sshd23:39:243
60499699624,18cyclictest687460-21modprobe23:55:001
604996996224,4cyclictest617858-21sh21:24:311
604996996219,8cyclictest348592-1kworker/u9:221:11:541
604996996219,8cyclictest348592-1kworker/u9:221:11:531
605003996119,37cyclictest275-21dbus-daemon20:40:003
60500399611,15cyclictest48-21kcompactd022:30:133
60500399611,15cyclictest48-21kcompactd022:30:123
60500399611,15cyclictest364-21snmpd23:18:523
60500399611,15cyclictest364-21snmpd23:18:513
60500399611,13cyclictest3467602-21systemd-journal22:42:073
60500399611,13cyclictest3467602-21systemd-journal22:42:063
60500399611,12cyclictest635079-21ssh22:01:283
60500399611,12cyclictest635079-21ssh22:01:273
604996996126,31cyclictest644331-21sh22:21:271
604996996126,31cyclictest644331-21sh22:21:261
604996996125,5cyclictest683131-21sh23:45:141
60499299618,48cyclictest16850irq/134-800000021:32:210
605003996022,35cyclictest37-21ksoftirqd/322:26:463
60500399602,16cyclictest678124-21latency_hist23:34:593
60500399602,16cyclictest668080-21ssh23:13:023
60500399602,13cyclictest626819-21ssh21:43:503
60500399602,13cyclictest364-21snmpd22:20:373
60500399602,13cyclictest364-21snmpd22:20:373
60500399601,14cyclictest609066-21sshd20:28:233
604996996027,29cyclictest654976-21sh22:44:441
604996996027,29cyclictest654976-21sh22:44:441
604996996025,11cyclictest23-21ksoftirqd/122:30:431
604996996025,11cyclictest23-21ksoftirqd/122:30:431
604996996016,9cyclictest649540-1kworker/u9:223:28:551
604996996016,9cyclictest649540-1kworker/u9:223:28:541
60500399591,13cyclictest364-21snmpd20:02:543
60500399591,13cyclictest364-21snmpd00:36:183
604996995926,29cyclictest657995-21sh22:50:591
604996995926,29cyclictest657995-21sh22:50:581
60500399582,17cyclictest607281-21latency_hist19:55:023
60500399582,16cyclictest617098-21ssh21:22:423
60500399582,15cyclictest665592-21ssh23:07:333
60500399582,14cyclictest624670-21ssh21:39:153
605000995830,25cyclictest291rcuc/223:15:022
605000995830,25cyclictest291rcuc/223:15:022
60499699586,14cyclictest0-21swapper/100:20:001
604996995828,27cyclictest23-21ksoftirqd/123:37:311
604996995826,10cyclictest23-21ksoftirqd/123:11:061
604996995823,12cyclictest1-21systemd21:50:001
604996995822,4cyclictest637819-21sh22:07:331
604992995828,8cyclictest12-21ksoftirqd/020:40:010
605003995727,23cyclictest612784-21sh21:13:233
605003995727,23cyclictest612784-21sh21:13:223
605003995717,36cyclictest628671-21kworker/u8:321:56:493
605003995716,32cyclictest0-21swapper/320:10:023
60500399571,17cyclictest610922-21munin-run21:05:013
60500399571,15cyclictest691983-21ssh00:04:583
60500399571,14cyclictest686513-21ssh23:52:493
60500399571,14cyclictest364-21snmpd22:36:113
60500399571,13cyclictest364-21snmpd19:14:293
605000995748,5cyclictest648146-21kworker/2:023:41:472
604996995722,4cyclictest676320-21sh23:30:381
604996995721,9cyclictest23-21ksoftirqd/122:25:291
604996995714,11cyclictest317166-1kworker/u9:022:20:001
604992995720,7cyclictest12-21ksoftirqd/019:37:500
605003995621,31cyclictest275-21dbus-daemon23:05:013
605003995621,31cyclictest275-21dbus-daemon23:05:013
60500399562,12cyclictest674821-21ssh23:27:333
60500399562,12cyclictest674821-21ssh23:27:323
605003995616,6cyclictest606374-21cat19:35:023
605003995616,6cyclictest606374-21cat19:35:023
60500399561,17cyclictest621264-21ssh21:31:323
60500399561,14cyclictest606514-21systemd-run19:40:003
605000995646,7cyclictest698879-21kworker/2:300:34:172
604996995626,7cyclictest23-21ksoftirqd/100:39:001
604996995624,10cyclictest23-21ksoftirqd/100:02:181
604996995619,4cyclictest705858-21sh00:35:031
60499299562,19cyclictest1-21systemd20:55:010
605003995522,30cyclictest37-21ksoftirqd/322:05:203
605003995516,34cyclictest608500-21latency_hist20:15:023
605000995547,5cyclictest618139-21kworker/2:122:16:012
605000995543,7cyclictest683270-21kworker/2:100:22:462
605000995510,26cyclictest1301767-1kworker/2:2H21:15:022
605000995510,26cyclictest1301767-1kworker/2:2H21:15:022
604996995523,8cyclictest23-21ksoftirqd/120:26:251
604996995520,5cyclictest608774-21latency_hist20:20:021
604992995523,7cyclictest12-21ksoftirqd/019:32:340
604992995523,7cyclictest12-21ksoftirqd/019:32:340
604992995517,33cyclictest16850irq/134-800000020:32:240
604992995517,33cyclictest16850irq/134-800000020:32:230
604992995514,10cyclictest275-21dbus-daemon20:20:000
605003995422,29cyclictest37-21ksoftirqd/321:18:273
60500399542,14cyclictest684898-21ssh23:49:333
60500399542,13cyclictest682823-21munin-run23:45:003
605003995416,32cyclictest309-21in:imuxsock19:25:003
60500399541,12cyclictest631822-21ssh21:54:513
605000995446,5cyclictest613480-21kworker/2:322:35:472
605000995445,5cyclictest618139-21kworker/2:121:36:182
605000995445,5cyclictest613480-21kworker/2:323:21:122
605000995444,6cyclictest613480-21kworker/2:322:47:152
604996995425,4cyclictest660345-21sh22:56:001
604996995423,9cyclictest23-21ksoftirqd/123:43:011
604996995423,4cyclictest611324-21latency_hist21:10:021
604996995422,4cyclictest609479-21latency_hist20:35:031
604996995422,4cyclictest609479-21latency_hist20:35:021
604992995422,7cyclictest12-21ksoftirqd/020:23:520
60500399532,6cyclictest131rcu_preempt20:20:003
605003995324,25cyclictest37-21ksoftirqd/319:30:013
60500399532,17cyclictest610129-21true20:50:013
60500399532,17cyclictest610129-21true20:50:013
60500399531,12cyclictest364-21snmpd20:55:283
605000995346,4cyclictest683270-21kworker/2:100:10:182
605000995345,5cyclictest618139-21kworker/2:122:02:202
605000995345,5cyclictest618139-21kworker/2:122:02:202
605000995345,5cyclictest618139-21kworker/2:121:51:222
605000995345,5cyclictest613480-21kworker/2:322:42:242
605000995345,5cyclictest613480-21kworker/2:322:42:242
605000995344,6cyclictest618139-21kworker/2:121:42:262
605000995344,5cyclictest613480-21kworker/2:321:23:372
605003995218,31cyclictest3467562-21sshd21:05:233
605000995242,6cyclictest683270-21kworker/2:100:25:462
604996995225,11cyclictest23-21ksoftirqd/123:02:581
604996995225,11cyclictest23-21ksoftirqd/123:02:581
604996995220,7cyclictest23-21ksoftirqd/121:52:411
605000995143,5cyclictest683270-21kworker/2:100:04:582
605000995143,5cyclictest648146-21kworker/2:023:51:232
605000995143,5cyclictest648146-21kworker/2:022:50:212
605000995143,5cyclictest648146-21kworker/2:022:50:202
605000995143,5cyclictest618139-21kworker/2:122:25:362
605000995143,5cyclictest618139-21kworker/2:121:58:352
605000995143,5cyclictest618139-21kworker/2:121:33:382
605000995142,6cyclictest618139-21kworker/2:121:49:492
605000995142,5cyclictest683270-21kworker/2:100:17:402
605000995141,6cyclictest618139-21kworker/2:122:23:562
605000995141,6cyclictest618139-21kworker/2:122:23:562
604996995122,10cyclictest23-21ksoftirqd/123:21:311
604996995121,7cyclictest23-21ksoftirqd/121:43:571
60500399502,12cyclictest364-21snmpd19:40:423
60500399501,14cyclictest1-21systemd20:15:013
60500099508,24cyclictest0-21swapper/219:20:022
605000995042,5cyclictest618139-21kworker/2:122:10:392
605000995042,5cyclictest618139-21kworker/2:122:08:342
605000995042,5cyclictest618139-21kworker/2:121:25:352
605000995042,5cyclictest613480-21kworker/2:323:00:352
605000995042,5cyclictest613480-21kworker/2:323:00:352
605000995042,5cyclictest609898-21kworker/2:021:15:262
605000995041,5cyclictest613480-21kworker/2:323:30:202
605000995040,6cyclictest698879-21kworker/2:300:38:102
60499299501,31cyclictest0-21swapper/020:03:020
60500399493,11cyclictest609337-21munin-run20:35:003
60500399493,11cyclictest609337-21munin-run20:35:003
605000994941,5cyclictest683270-21kworker/2:100:05:242
605000994941,5cyclictest648146-21kworker/2:023:55:322
605000994941,5cyclictest648146-21kworker/2:023:05:042
605000994941,5cyclictest618139-21kworker/2:122:30:102
605000994941,5cyclictest618139-21kworker/2:122:30:102
605000994941,5cyclictest613480-21kworker/2:323:35:142
605000994941,5cyclictest613480-21kworker/2:322:56:442
605000994941,5cyclictest604201-21kworker/2:319:35:292
605000994940,6cyclictest648146-21kworker/2:023:45:142
605000994940,6cyclictest613480-21kworker/2:323:25:232
605000994940,6cyclictest613480-21kworker/2:323:25:222
605000994940,5cyclictest608865-21kworker/2:221:01:232
604996994915,11cyclictest23-21ksoftirqd/100:30:001
60499299491,29cyclictest0-21swapper/020:28:230
60499299491,12cyclictest606750-21cron19:45:000
60500399482,13cyclictest364-21snmpd20:41:133
605000994840,5cyclictest648146-21kworker/2:023:10:412
605000994840,5cyclictest608865-21kworker/2:220:42:232
605000994839,6cyclictest604201-21kworker/2:320:27:332
60500099482,41cyclictest610124-21grep20:50:012
60500099482,41cyclictest610124-21grep20:50:002
604996994822,8cyclictest23-21ksoftirqd/123:08:561
60499299481,4cyclictest0-21swapper/019:51:500
605000994739,5cyclictest608865-21kworker/2:221:05:242
605000994739,5cyclictest604201-21kworker/2:320:23:042
605000994739,5cyclictest604201-21kworker/2:320:05:432
605000994739,5cyclictest604201-21kworker/2:319:41:002
605000994739,5cyclictest604201-21kworker/2:319:20:452
605000994733,9cyclictest3467602-21systemd-journal20:11:232
605000994733,9cyclictest3467602-21systemd-journal20:11:232
60499699472,6cyclictest131rcu_preempt23:20:011
60499699472,6cyclictest131rcu_preempt23:20:001
604996994720,7cyclictest23-21ksoftirqd/120:05:021
604996994719,4cyclictest606222-21latency_hist19:35:021
604996994719,4cyclictest606222-21latency_hist19:35:021
60499299471,26cyclictest0-21swapper/019:48:480
60499299471,26cyclictest0-21swapper/019:48:470
60499299471,26cyclictest0-21swapper/019:11:550
605000994638,5cyclictest608865-21kworker/2:220:53:172
605000994638,5cyclictest608865-21kworker/2:220:35:342
605000994638,5cyclictest604201-21kworker/2:319:26:102
605000994637,5cyclictest604201-21kworker/2:320:18:332
60500399454,7cyclictest0-21swapper/319:20:023
605000994537,5cyclictest608865-21kworker/2:220:58:322
605000994537,5cyclictest604201-21kworker/2:320:01:382
605000994537,5cyclictest604201-21kworker/2:319:57:132
605000994537,5cyclictest604201-21kworker/2:319:57:132
605000994537,5cyclictest604201-21kworker/2:319:51:242
605000994536,6cyclictest604201-21kworker/2:320:30:242
605000994536,6cyclictest604201-21kworker/2:320:30:242
60499699457,5cyclictest131rcu_preempt00:05:151
605000994436,5cyclictest604201-21kworker/2:319:45:082
605000994436,5cyclictest604201-21kworker/2:319:45:082
604996994427,13cyclictest23-21ksoftirqd/119:30:001
60499699441,4cyclictest131rcu_preempt19:21:481
60500399432,13cyclictest605021-21cat19:10:023
605000994331,8cyclictest3467602-21systemd-journal19:11:242
604996994329,10cyclictest309-21in:imuxsock20:15:011
604996994329,10cyclictest309-21in:imuxsock20:15:011
60499299431,38cyclictest605448-21systemctl19:20:000
605000994231,7cyclictest0-21swapper/219:10:022
60499699424,33cyclictest150-21pr/ttyS219:15:471
604996994228,11cyclictest610888-21sshd21:03:231
604992994235,4cyclictest16950irq/224-800000020:50:000
604992994235,4cyclictest16950irq/224-800000020:50:000
60499299422,6cyclictest608280-21cron20:15:000
60499299422,6cyclictest608280-21cron20:15:000
605000994132,6cyclictest604201-21kworker/2:319:31:352
605000994132,6cyclictest604201-21kworker/2:319:31:352
604996994130,7cyclictest23-21ksoftirqd/121:30:001
60499699412,3cyclictest131rcu_preempt20:20:021
60499699407,5cyclictest131rcu_preempt21:32:171
60499699403,4cyclictest131rcu_preempt00:24:211
60499699403,3cyclictest131rcu_preempt21:16:131
604992994033,4cyclictest16950irq/224-800000021:00:010
60499699394,3cyclictest131rcu_preempt20:06:231
604996993925,10cyclictest275-21dbus-daemon19:40:001
60499699392,33cyclictest150-21pr/ttyS219:44:521
604992993926,10cyclictest3467602-21systemd-journal21:05:010
604992993913,20cyclictest605714-21modprobe19:25:000
604996993828,7cyclictest23-21ksoftirqd/120:58:241
60499699375,27cyclictest610050-21wc20:45:021
604992993726,7cyclictest12-21ksoftirqd/020:45:010
604992993725,9cyclictest12-21ksoftirqd/019:10:020
604996993626,7cyclictest607232-21sshd19:51:491
604996993625,8cyclictest610312-21cat20:50:021
60499699361,3cyclictest131rcu_preempt19:10:351
604996993527,5cyclictest23-21ksoftirqd/120:00:001
604996993527,5cyclictest23-21ksoftirqd/120:00:001
604996993418,12cyclictest3467602-21systemd-journal19:50:001
604996993418,12cyclictest3467602-21systemd-journal19:50:001
604996993323,7cyclictest610280-21cat20:50:031
604996992515,7cyclictest605022-21date19:10:031
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional