You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-23 - 14:32

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Sun Aug 23, 2026 00:45:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
351597399652,12cyclictest3575295-21sh23:03:491
351597399651,14cyclictest3579856-21ssh23:13:281
3515973996511,11cyclictest3518661-21cat20:00:021
351597399642,12cyclictest3578071-21ssh23:09:531
3515973996417,40cyclictest309-21in:imuxsock23:20:551
351597399641,13cyclictest3532504-21ssh21:32:101
351597399641,12cyclictest3600536-21ssh23:57:381
3515968996456,6cyclictest12-21ksoftirqd/021:40:030
3515968996456,6cyclictest12-21ksoftirqd/021:40:020
3515968996442,18cyclictest16850irq/134-800000022:00:000
3515968996414,45cyclictest0-21swapper/022:35:000
351597399632,27cyclictest3571317-21sh22:55:021
3515973996322,35cyclictest309-21in:imuxsock20:30:001
351597399632,14cyclictest3549999-21munin-run22:10:001
351597399631,15cyclictest3531315-21sshd21:29:551
351597399631,15cyclictest3531315-21sshd21:29:551
351597399631,13cyclictest3596410-21ssh23:48:531
351597399631,13cyclictest3562096-21ssh22:35:221
351597399631,13cyclictest3547081-21ssh22:03:401
351597399631,12cyclictest3581842-21ssh23:17:421
3515973996243,6cyclictest23-21ksoftirqd/121:20:031
3515973996243,6cyclictest23-21ksoftirqd/121:20:031
3515973996219,38cyclictest3516935-21cron19:30:001
351596899627,37cyclictest131rcu_preempt21:33:090
3515968996213,6cyclictest131rcu_preempt00:00:020
3515973996125,31cyclictest3573418-21sendmail23:00:001
351597399612,13cyclictest3591900-21ssh23:39:241
351597399612,13cyclictest3591900-21ssh23:39:241
3515973996115,40cyclictest23-21ksoftirqd/123:54:161
351597399611,13cyclictest3467602-21systemd-journal21:40:211
351597899605,37cyclictest3601443-40mandb00:00:023
351597399602,17cyclictest3526242-21ssh21:18:451
351597399602,15cyclictest3585868-21ssh23:26:031
351597399602,14cyclictest3522171-21munin-run21:10:011
351597399602,13cyclictest3518239-21munin-run19:55:001
351597399601,14cyclictest364-21snmpd23:42:201
351597399601,13cyclictest3557664-21sshd22:25:551
351597399601,13cyclictest3557664-21sshd22:25:551
351597399601,13cyclictest3519003-21sshd20:09:551
351597399601,11cyclictest3517442-21cron19:40:001
3515968996024,32cyclictest12-21ksoftirqd/022:43:210
3515973995915,38cyclictest23-21ksoftirqd/120:20:001
351597399591,14cyclictest48-21kcompactd000:00:371
351597399591,13cyclictest3611553-21ssh00:21:351
351597399591,13cyclictest3611553-21ssh00:21:341
351597399591,12cyclictest3544068-21ssh21:56:581
3515968995925,31cyclictest12-21ksoftirqd/000:03:240
3515973995833,21cyclictest3588319-21rm23:31:241
351597399582,20cyclictest3566978-21ssh22:45:501
351597399582,15cyclictest3541736-21ssh21:51:571
351597399582,14cyclictest3606211-21wc00:10:021
351597399582,12cyclictest3517172-21sshd19:31:541
351597399582,12cyclictest3517172-21sshd19:31:541
351597399582,11cyclictest364-21snmpd22:22:531
351597399581,22cyclictest3536080-21sendmail-msp21:40:001
351597399581,22cyclictest3536080-21sendmail-msp21:40:001
351597399581,13cyclictest3560030-21ssh22:30:591
3515968995833,15cyclictest3467602-21systemd-journal23:10:010
3515968995830,25cyclictest3609163-21ssh00:16:270
3515968995829,13cyclictest3575033-21ssh23:03:150
3515968995827,25cyclictest12-21ksoftirqd/000:15:010
3515968995825,29cyclictest12-21ksoftirqd/021:42:190
3515968995819,34cyclictest3615194-21sshd00:29:430
3515978995747,6cyclictest3568801-21grep22:50:003
3515975995735,16cyclictest1-21systemd21:55:012
351597399578,3cyclictest0-21swapper/119:12:501
351597399572,13cyclictest364-21snmpd21:12:051
351597399572,11cyclictest3517714-21sshd19:43:551
3515973995711,8cyclictest3547466-1kworker/u9:122:40:171
351597399571,16cyclictest3615323-21ssh00:29:571
3515968995729,24cyclictest12-21ksoftirqd/021:46:550
3515968995726,26cyclictest3545970-21ssh22:00:510
3515968995723,30cyclictest12-21ksoftirqd/022:13:040
3515975995648,5cyclictest3519048-21kworker/2:000:20:312
3515975995648,5cyclictest3519048-21kworker/2:000:20:302
3515975995646,6cyclictest3519048-21kworker/2:000:25:412
3515975995645,6cyclictest3519048-21kworker/2:021:30:002
3515975995645,6cyclictest3519048-21kworker/2:021:30:002
351597399562,14cyclictest3554632-21ssh22:19:501
351597399562,14cyclictest3539507-21ssh21:47:101
351597399562,12cyclictest3550990-21ssh22:11:491
351597399561,12cyclictest364-21snmpd20:41:191
351596899568,4cyclictest131rcu_preempt21:27:270
351596899568,4cyclictest131rcu_preempt21:27:270
3515968995627,13cyclictest3619468-21ssh00:38:480
3515975995546,6cyclictest3519048-21kworker/2:021:22:422
3515975995546,6cyclictest3519048-21kworker/2:021:22:412
3515975995546,5cyclictest3519048-21kworker/2:000:05:012
3515975995543,7cyclictest3519048-21kworker/2:023:25:522
351597399552,13cyclictest3609693-21ssh00:17:321
351597399552,12cyclictest3617689-21munin-run00:35:021
351597399552,12cyclictest3617689-21munin-run00:35:011
351597399552,11cyclictest3618932-21ssh00:37:351
3515973995517,34cyclictest3516459-21systemd-run19:20:001
3515968995523,28cyclictest12-21ksoftirqd/023:45:020
351596899552,31cyclictest131rcu_preempt20:25:000
351596899552,31cyclictest131rcu_preempt20:25:000
3515975995445,6cyclictest3519048-21kworker/2:023:56:302
3515975995445,6cyclictest3519048-21kworker/2:023:01:012
351597399545,10cyclictest3521607-21modprobe21:00:001
351597399542,14cyclictest3519286-21munin-run20:15:011
3515973995420,28cyclictest3520740-21wc20:40:021
3515973995420,28cyclictest3520740-21wc20:40:021
351596899549,37cyclictest3516691-21modprobe19:25:000
351596899543,11cyclictest3518476-21cron20:00:000
3515968995426,12cyclictest3591393-21ssh23:38:100
3515968995426,12cyclictest3591393-21ssh23:38:090
3515968995426,12cyclictest3583247-21ssh23:20:310
3515975995346,4cyclictest3519048-21kworker/2:023:20:122
3515975995345,5cyclictest3519048-21kworker/2:021:38:142
3515975995345,5cyclictest3519048-21kworker/2:021:38:142
3515975995344,6cyclictest3519048-21kworker/2:021:12:342
3515975995343,6cyclictest3519048-21kworker/2:000:05:172
351597399533,9cyclictest3521877-21cron21:05:001
351597399531,12cyclictest3607711-21ssh00:13:261
3515968995327,11cyclictest3569199-21ssh22:50:340
3515968995324,14cyclictest3556313-21ssh22:23:130
3515968995315,7cyclictest12-21ksoftirqd/023:30:000
3515975995244,5cyclictest3519048-21kworker/2:023:20:032
3515975995243,6cyclictest3519048-21kworker/2:022:42:512
3515975995243,5cyclictest3519048-21kworker/2:000:20:002
3515975995241,6cyclictest3519048-21kworker/2:000:31:472
3515975995241,6cyclictest3519048-21kworker/2:000:31:462
351597399522,11cyclictest3516685-21cron19:25:001
351597399521,12cyclictest364-21snmpd20:33:581
3515968995225,12cyclictest3588082-21ssh23:30:430
3515968995218,30cyclictest12-21ksoftirqd/021:12:550
3515975995143,5cyclictest3519048-21kworker/2:023:47:232
3515975995143,5cyclictest3519048-21kworker/2:022:25:202
3515975995143,5cyclictest3519048-21kworker/2:022:25:202
3515975995143,5cyclictest3519048-21kworker/2:022:03:272
3515975995143,5cyclictest3519048-21kworker/2:021:58:532
3515975995142,6cyclictest3519048-21kworker/2:023:39:582
3515975995142,6cyclictest3519048-21kworker/2:023:39:582
3515975995142,5cyclictest3519048-21kworker/2:023:08:032
3515975995142,5cyclictest3519048-21kworker/2:021:20:012
3515975995141,6cyclictest3519048-21kworker/2:022:57:432
3515975995141,6cyclictest3519048-21kworker/2:022:50:452
3515975995141,6cyclictest3519048-21kworker/2:022:08:342
3515975995140,6cyclictest3491256-21kworker/2:220:50:002
351597399512,11cyclictest3521243-21cat20:50:031
351596899513,9cyclictest3517195-21modprobe19:35:000
351596899513,9cyclictest3517195-21modprobe19:35:000
3515968995128,19cyclictest3558359-21ssh22:27:320
3515968995128,19cyclictest3558359-21ssh22:27:320
3515968995127,5cyclictest131rcu_preempt21:22:430
3515968995127,5cyclictest131rcu_preempt21:22:430
3515968995125,11cyclictest3563132-21ssh22:37:570
3515968995124,23cyclictest12-21ksoftirqd/023:19:040
3515968995124,12cyclictest3525127-21ssh21:16:180
3515978995030,15cyclictest3536053-21cron21:40:003
3515978995030,15cyclictest3536053-21cron21:40:003
3515975995042,5cyclictest3519048-21kworker/2:023:52:422
3515975995042,5cyclictest3519048-21kworker/2:023:34:012
3515975995042,5cyclictest3519048-21kworker/2:023:10:212
3515975995042,5cyclictest3519048-21kworker/2:022:49:222
3515975995042,5cyclictest3519048-21kworker/2:022:31:022
3515975995042,5cyclictest3519048-21kworker/2:021:05:022
3515975995042,5cyclictest3519048-21kworker/2:000:39:412
351596899506,37cyclictest3517468-21sendmail19:40:000
3515968995043,4cyclictest16950irq/224-800000020:50:010
3515968995038,8cyclictest12-21ksoftirqd/022:18:210
3515968995025,11cyclictest3548724-21ssh22:06:590
3515968995023,23cyclictest12-21ksoftirqd/022:47:490
3515968995017,29cyclictest12-21ksoftirqd/000:09:450
3515978994931,13cyclictest275-21dbus-daemon21:20:013
3515975994941,5cyclictest3519048-21kworker/2:022:19:172
3515975994941,5cyclictest3519048-21kworker/2:022:11:102
3515975994941,5cyclictest3519048-21kworker/2:000:12:402
3515975994940,6cyclictest3519048-21kworker/2:021:43:522
3515975994940,5cyclictest3491256-21kworker/2:220:00:022
3515973994912,11cyclictest23-21ksoftirqd/120:04:341
3515968994930,15cyclictest3573560-21ssh23:00:020
3515968994919,14cyclictest3541448-21ssh21:51:090
3515968994916,7cyclictest12-21ksoftirqd/023:10:360
3515975994841,5cyclictest3519048-21kworker/2:021:33:162
3515975994840,5cyclictest3519048-21kworker/2:023:40:592
3515975994840,5cyclictest3491256-21kworker/2:220:45:022
3515975994840,5cyclictest3491256-21kworker/2:220:27:552
3515975994839,6cyclictest3519048-21kworker/2:022:38:062
3515975994839,6cyclictest3491256-21kworker/2:220:25:022
3515975994839,6cyclictest3491256-21kworker/2:220:25:022
3515975994839,5cyclictest3491256-21kworker/2:219:30:012
3515975994838,5cyclictest3491256-21kworker/2:219:50:002
351597399482,15cyclictest3467602-21systemd-journal19:50:001
3515968994831,14cyclictest3596171-21ssh23:48:220
3515968994830,13cyclictest12-21ksoftirqd/019:50:000
3515968994821,12cyclictest3598226-21ssh23:52:460
3515968994819,25cyclictest12-21ksoftirqd/019:25:490
3515978994729,13cyclictest3467602-21systemd-journal22:10:023
3515975994739,5cyclictest3519048-21kworker/2:022:20:232
3515975994738,6cyclictest3519048-21kworker/2:021:45:412
3515975994738,5cyclictest3491256-21kworker/2:219:30:502
3515975994738,5cyclictest3491256-21kworker/2:219:30:502
3515968994728,10cyclictest12-21ksoftirqd/021:10:000
351597899464,3cyclictest131rcu_preempt21:30:013
351597899464,3cyclictest131rcu_preempt21:30:013
351597899462,4cyclictest131rcu_preempt00:18:203
351597899462,3cyclictest131rcu_preempt23:34:323
3515978994621,6cyclictest37-21ksoftirqd/321:32:103
3515978994620,4cyclictest37-21ksoftirqd/322:40:013
3515975994637,6cyclictest3491256-21kworker/2:220:10:012
3515968994638,5cyclictest16950irq/224-800000019:15:010
3515968994619,13cyclictest3611479-21ssh00:21:240
3515968994619,13cyclictest3611479-21ssh00:21:240
351597899457,6cyclictest131rcu_preempt23:51:433
351597899454,3cyclictest150-21pr/ttyS222:29:093
351597899454,3cyclictest150-21pr/ttyS222:29:093
351597899454,3cyclictest131rcu_preempt00:07:433
351597899453,4cyclictest131rcu_preempt00:11:563
3515978994534,7cyclictest1-21systemd19:20:003
3515978994531,11cyclictest37-21ksoftirqd/322:54:093
3515978994530,12cyclictest275-21dbus-daemon22:55:493
3515978994529,12cyclictest37-21ksoftirqd/323:29:233
351597899451,3cyclictest131rcu_preempt21:41:573
3515978994510,6cyclictest131rcu_preempt00:05:003
3515975994537,5cyclictest3519048-21kworker/2:021:10:032
3515975994537,5cyclictest3491256-21kworker/2:219:20:552
3515973994516,9cyclictest23-21ksoftirqd/120:22:081
3515973994516,9cyclictest23-21ksoftirqd/120:22:081
351596899453,7cyclictest12-21ksoftirqd/020:10:000
351597899443,4cyclictest131rcu_preempt22:34:233
351597899443,3cyclictest131rcu_preempt22:20:403
351597899443,3cyclictest131rcu_preempt00:36:123
3515978994432,9cyclictest37-21ksoftirqd/321:56:333
3515978994425,14cyclictest3520576-21cat20:40:003
3515978994425,14cyclictest3520576-21cat20:40:003
3515978994417,5cyclictest37-21ksoftirqd/323:20:033
351597899441,4cyclictest131rcu_preempt23:45:003
3515975994435,6cyclictest3491256-21kworker/2:219:55:012
3515973994428,11cyclictest275-21dbus-daemon20:55:011
3515968994431,9cyclictest12-21ksoftirqd/021:00:010
351597899433,4cyclictest131rcu_preempt00:33:453
351597899433,4cyclictest131rcu_preempt00:33:453
3515978994321,17cyclictest275-21dbus-daemon23:05:013
3515975994336,4cyclictest3491256-21kworker/2:219:11:202
3515975994335,4cyclictest3491256-21kworker/2:221:00:012
3515975994334,5cyclictest3491256-21kworker/2:220:15:012
3515968994327,12cyclictest12-21ksoftirqd/019:10:020
351596899432,38cyclictest3615633-21ssh00:30:080
351596899432,38cyclictest3615633-21ssh00:30:080
351597899426,3cyclictest150-21pr/ttyS221:50:013
351597899422,7cyclictest0-21swapper/322:45:003
3515978994219,4cyclictest37-21ksoftirqd/321:15:013
3515978994219,4cyclictest37-21ksoftirqd/319:55:013
3515978994218,6cyclictest37-21ksoftirqd/323:14:253
3515975994226,11cyclictest275-21dbus-daemon20:05:002
351597899416,3cyclictest131rcu_preempt21:25:013
351597899416,3cyclictest131rcu_preempt21:25:003
3515978994127,12cyclictest37-21ksoftirqd/322:19:333
3515978994124,13cyclictest3467602-21systemd-journal20:05:003
351597899412,3cyclictest131rcu_preempt23:21:503
3515978994117,6cyclictest37-21ksoftirqd/322:04:403
3515978994117,5cyclictest37-21ksoftirqd/321:50:413
351597899411,4cyclictest131rcu_preempt22:07:413
3515975994133,5cyclictest3491256-21kworker/2:220:15:542
3515968994132,5cyclictest309-21in:imuxsock19:20:000
3515968994130,9cyclictest12-21ksoftirqd/020:40:020
3515968994130,9cyclictest12-21ksoftirqd/020:40:020
3515968994129,8cyclictest3518764-21cron20:05:000
3515968994128,10cyclictest12-21ksoftirqd/020:35:000
351596899412,33cyclictest3519465-21latency_hist20:15:030
351596899411,35cyclictest151rcuc/021:05:010
351597899406,9cyclictest37-21ksoftirqd/300:29:213
351597899404,4cyclictest131rcu_preempt23:36:303
351597899404,4cyclictest131rcu_preempt23:36:303
3515978994028,8cyclictest37-21ksoftirqd/321:10:003
3515978994016,6cyclictest37-21ksoftirqd/323:06:263
351596899408,10cyclictest12-21ksoftirqd/019:51:300
3515968994033,5cyclictest12-21ksoftirqd/020:20:000
351597899394,3cyclictest150-21pr/ttyS200:24:023
351597899394,3cyclictest150-21pr/ttyS200:24:023
3515975993930,6cyclictest3491256-21kworker/2:220:55:012
3515975993925,10cyclictest275-21dbus-daemon20:35:012
351597399391,11cyclictest3516053-21latency_hist19:10:021
351596899398,9cyclictest12-21ksoftirqd/020:41:200
351597899383,3cyclictest131rcu_preempt23:48:323
351597899383,3cyclictest131rcu_preempt20:20:003
3515978993825,8cyclictest309-21in:imuxsock20:34:553
3515978993818,5cyclictest37-21ksoftirqd/319:40:003
3515978993817,6cyclictest361rcuc/320:50:023
3515975993830,5cyclictest3491256-21kworker/2:219:36:552
3515975993820,13cyclictest275-21dbus-daemon19:20:002
351596899387,6cyclictest131rcu_preempt20:30:000
351597899374,5cyclictest131rcu_preempt20:10:003
3515978993724,10cyclictest37-21ksoftirqd/320:15:023
351597899372,30cyclictest275-21dbus-daemon19:45:013
351597899371,3cyclictest131rcu_preempt19:25:013
351596899378,5cyclictest131rcu_preempt20:50:550
351597899362,3cyclictest131rcu_preempt19:15:013
3515978993616,4cyclictest37-21ksoftirqd/320:00:003
351597599363,3cyclictest131rcu_preempt19:45:002
3515975993628,5cyclictest3491256-21kworker/2:220:36:432
3515975993628,5cyclictest3491256-21kworker/2:220:36:432
351597899353,3cyclictest131rcu_preempt19:35:033
351597899353,3cyclictest131rcu_preempt19:35:033
3515978993522,9cyclictest37-21ksoftirqd/321:05:013
3515978993515,5cyclictest37-21ksoftirqd/319:30:013
351597899351,3cyclictest131rcu_preempt21:00:003
351596899351,31cyclictest0-21swapper/019:41:000
351597899343,3cyclictest131rcu_preempt20:55:013
3515978993416,5cyclictest37-21ksoftirqd/320:30:013
351597899341,4cyclictest131rcu_preempt20:45:013
351597899322,3cyclictest131rcu_preempt20:21:013
351597899322,3cyclictest131rcu_preempt20:21:013
351597899312,3cyclictest131rcu_preempt19:45:033
3515975993024,4cyclictest30-21ksoftirqd/219:10:022
3515978992822,4cyclictest3515980-21latency_hist19:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional