You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-02 - 07:51

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Thu Apr 02, 2026 00:45:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
405949899812,43cyclictest4086346-21sh21:57:431
405949899812,40cyclictest4143520-21systemd-run00:10:011
405949899804,37cyclictest4153100-21ssh00:31:571
405949899804,37cyclictest4153100-21ssh00:31:561
405949899791,41cyclictest4133961-21ssh23:47:431
405949899781,43cyclictest4090336-21ssh22:06:521
405949899771,40cyclictest4081397-21ssh21:45:591
405949899762,38cyclictest4123739-21ssh23:24:211
405949899761,42cyclictest4149655-21rm00:24:101
405949899761,38cyclictest4076116-21ssh21:34:111
405949899753,38cyclictest4078077-21ssh21:38:371
405949899751,42cyclictest4064921-21cut20:55:011
405949899722,38cyclictest4105041-21ssh22:40:521
405949899722,38cyclictest4105041-21ssh22:40:511
405949899722,37cyclictest4112083-21ssh22:57:131
405949899721,36cyclictest4088832-21ssh22:03:381
405949899721,30cyclictest4118305-21ssh23:11:331
4059496997125,9cyclictest4072511-21rm21:25:230
405949899691,64cyclictest4093266-21ssh22:13:541
405949899681,43cyclictest4125197-21ssh23:27:351
405949899681,37cyclictest4073020-21ssh21:26:451
405949899681,35cyclictest4147627-21ssh00:19:361
405949699682,46cyclictest12-21ksoftirqd/021:45:340
405949899673,36cyclictest4144759-21sh00:12:421
405949899671,62cyclictest4126510-21ssh23:30:191
405949899671,34cyclictest4103512-21ssh22:37:271
405949899663,31cyclictest4100787-21ssh22:30:541
405949899663,31cyclictest4100787-21ssh22:30:531
4059498996627,35cyclictest4155285-21sh00:37:021
405949899662,60cyclictest4096328-21ssh22:20:361
405949899662,59cyclictest4061268-21munin-run19:45:011
4059498996622,40cyclictest4109783-21sh22:51:551
405949899661,62cyclictest4121531-21ssh23:19:151
405949899652,59cyclictest4117006-21ssh23:08:401
405949899652,58cyclictest4070481-21ssh21:20:441
4059498996525,36cyclictest4136908-21sh23:54:421
405949899651,61cyclictest4107733-21ssh22:47:071
405949899651,60cyclictest4099381-21ssh22:27:541
405949899651,60cyclictest4084826-21ssh21:54:271
405949899651,60cyclictest4079333-21ssh21:41:201
405949899651,60cyclictest4068978-21ssh21:17:271
405949899651,59cyclictest4139454-21ssh00:00:071
405949899641,59cyclictest4131151-21ssh23:40:591
405949899641,59cyclictest4094204-21ssh22:15:451
4059496996429,31cyclictest4121824-21latency_hist23:20:010
4059501996312,7cyclictest131rcu_preempt21:20:012
405949899631,58cyclictest4115214-21ssh23:04:411
405949899631,58cyclictest4066607-21ssh21:11:561
405949899631,57cyclictest3467602-21systemd-journal23:35:371
4059496996325,31cyclictest4154351-21latency_hist00:35:010
4059496996325,31cyclictest4154351-21latency_hist00:35:010
4059504996238,16cyclictest37-21ksoftirqd/322:34:503
4059504996238,16cyclictest37-21ksoftirqd/322:34:503
405949899622,55cyclictest4063587-21munin-run20:30:011
405949899621,58cyclictest4137747-21ssh23:56:201
405949899621,57cyclictest4150998-21ssh00:27:061
405949899621,55cyclictest48-21kcompactd020:17:401
405949899621,29cyclictest4063845-21systemd-run20:35:011
4059496996226,30cyclictest4065667-21latency_hist21:10:010
405949899612,55cyclictest4063300-21sshd20:20:251
4059496996123,29cyclictest16850irq/134-800000022:13:260
4059496996118,15cyclictest4135310-21kworker/u8:223:57:540
405949899601,53cyclictest48-21kcompactd019:37:111
405949899601,51cyclictest4065690-21ssh21:10:011
405949699602,24cyclictest4102636-21rm22:35:060
4059496996020,31cyclictest16850irq/134-800000021:54:410
4059496996019,34cyclictest16850irq/134-800000000:10:080
4059496996019,31cyclictest16850irq/134-800000023:35:470
405950499597,12cyclictest37-21ksoftirqd/300:00:003
405949899591,54cyclictest4060231-21munin-run19:25:001
4059496995921,33cyclictest12-21ksoftirqd/023:21:190
4059496995919,30cyclictest16850irq/134-800000023:42:190
4059496995917,33cyclictest16850irq/134-800000022:17:540
4059496995911,10cyclictest12-21ksoftirqd/022:29:540
405950499589,6cyclictest131rcu_preempt22:44:353
405950499589,6cyclictest131rcu_preempt22:44:353
405949899582,51cyclictest3467602-21systemd-journal21:04:301
405949899581,52cyclictest364-21snmpd19:46:031
405949899581,52cyclictest3467602-21systemd-journal19:10:451
405949699586,13cyclictest4097196-21rm22:22:460
405949699585,30cyclictest12-21ksoftirqd/022:43:570
405949699585,30cyclictest12-21ksoftirqd/022:43:560
4059496995830,13cyclictest4091240-21ssh22:09:190
4059504995710,6cyclictest131rcu_preempt23:13:563
405949899572,51cyclictest4065138-21munin-run21:00:021
405949899572,50cyclictest4062226-21date20:00:021
405949899571,52cyclictest4062776-21sshd20:14:251
405949899571,52cyclictest4062776-21sshd20:14:251
4059496995724,4cyclictest4080770-21latency_hist21:45:020
405949699572,39cyclictest4115603-21ssh23:05:030
4059496995718,30cyclictest16850irq/134-800000022:53:200
4059496995715,8cyclictest12-21ksoftirqd/023:53:110
4059504995646,8cyclictest361rcuc/322:00:003
405949899561,50cyclictest48-21kcompactd020:47:041
405949899561,50cyclictest48-21kcompactd020:40:581
405949899561,50cyclictest48-21kcompactd019:52:431
405949899561,50cyclictest48-21kcompactd019:16:381
4059496995620,28cyclictest16850irq/134-800000021:38:560
4059496995615,30cyclictest16850irq/134-800000021:17:280
405949699561,24cyclictest311-21rs:main0
4059496995611,9cyclictest12-21ksoftirqd/000:09:010
4059504995512,9cyclictest37-21ksoftirqd/322:48:153
405949899553,48cyclictest364-21snmpd20:06:041
405949899552,48cyclictest4060489-21latency_hist19:30:011
4059496995521,30cyclictest12-21ksoftirqd/023:11:420
4059496995520,26cyclictest16850irq/134-800000022:30:130
4059496995520,26cyclictest16850irq/134-800000022:30:120
405950499548,6cyclictest131rcu_preempt22:02:033
4059504995414,9cyclictest37-21ksoftirqd/322:40:023
4059501995446,5cyclictest4064589-21kworker/2:121:31:482
405949899541,49cyclictest48-21kcompactd020:04:541
4059496995425,13cyclictest4088774-21ssh22:03:260
4059496995419,31cyclictest12-21ksoftirqd/021:25:020
405949699541,37cyclictest4127254-21ssh23:32:040
405950499538,8cyclictest37-21ksoftirqd/321:42:223
4059504995315,6cyclictest131rcu_preempt00:09:223
4059504995314,9cyclictest131rcu_preempt00:27:573
4059504995313,6cyclictest131rcu_preempt19:30:013
4059501995314,8cyclictest30-21ksoftirqd/223:23:492
4059501995313,7cyclictest30-21ksoftirqd/223:51:302
405949699539,9cyclictest12-21ksoftirqd/020:15:020
405949699539,9cyclictest12-21ksoftirqd/020:15:010
405949699538,8cyclictest12-21ksoftirqd/000:19:310
405950499529,6cyclictest131rcu_preempt00:03:233
405950499524,8cyclictest37-21ksoftirqd/321:28:263
4059504995213,6cyclictest131rcu_preempt00:10:113
405950199529,4cyclictest131rcu_preempt00:31:212
405950199529,4cyclictest131rcu_preempt00:31:202
4059501995244,5cyclictest4128409-21kworker/2:200:19:552
4059501995244,5cyclictest4113233-21kworker/2:023:16:472
4059501995244,5cyclictest4113233-21kworker/2:023:03:562
4059501995244,5cyclictest4095936-21kworker/2:322:34:322
4059501995244,5cyclictest4095936-21kworker/2:322:34:312
4059501995244,5cyclictest4076490-21kworker/2:222:28:202
4059501995243,5cyclictest4128409-21kworker/2:200:22:472
4059501995223,12cyclictest30-21ksoftirqd/222:04:332
4059501995219,9cyclictest30-21ksoftirqd/222:45:262
4059501995211,6cyclictest131rcu_preempt20:20:012
405949699527,9cyclictest12-21ksoftirqd/000:35:330
4059496995213,5cyclictest0-21swapper/000:28:560
405950499515,7cyclictest131rcu_preempt21:38:003
4059504995132,8cyclictest37-21ksoftirqd/322:51:313
405950499511,8cyclictest0-21swapper/323:07:123
4059504995111,6cyclictest131rcu_preempt23:45:013
4059504995111,6cyclictest131rcu_preempt00:18:323
4059504995110,5cyclictest131rcu_preempt22:05:443
4059504995110,5cyclictest131rcu_preempt21:32:103
4059501995143,5cyclictest4104587-21kworker/2:123:41:232
4059501995143,5cyclictest4076490-21kworker/2:222:23:322
4059501995143,5cyclictest4064589-21kworker/2:122:13:282
4059501995143,5cyclictest4064589-21kworker/2:121:55:002
4059501995143,5cyclictest4064589-21kworker/2:121:45:092
4059501995143,5cyclictest4059707-21kworker/2:321:28:302
4059501995143,5cyclictest4059707-21kworker/2:321:11:222
4059501995142,5cyclictest4104587-21kworker/2:123:38:292
4059501995124,9cyclictest30-21ksoftirqd/200:05:002
405949899512,43cyclictest3467602-21systemd-journal19:30:021
405949899512,43cyclictest3467602-21systemd-journal19:30:021
405949899511,45cyclictest364-21snmpd20:36:321
4059496995123,12cyclictest4107587-21ssh22:46:460
405950499509,5cyclictest131rcu_preempt22:30:003
405950499507,8cyclictest37-21ksoftirqd/323:47:503
405950499507,12cyclictest37-21ksoftirqd/321:14:173
405950499504,8cyclictest37-21ksoftirqd/322:15:543
405950499502,11cyclictest37-21ksoftirqd/321:50:283
4059504995012,6cyclictest131rcu_preempt23:22:523
4059504995012,6cyclictest131rcu_preempt23:03:453
4059504995012,6cyclictest131rcu_preempt21:49:313
4059501995043,5cyclictest4128409-21kworker/2:200:29:342
4059501995043,4cyclictest4128409-21kworker/2:223:55:232
4059501995043,4cyclictest4076490-21kworker/2:222:42:132
4059501995043,4cyclictest4076490-21kworker/2:222:42:122
4059501995042,5cyclictest4128409-21kworker/2:200:38:232
4059501995042,5cyclictest4128409-21kworker/2:200:10:432
4059501995042,5cyclictest4128409-21kworker/2:200:09:302
4059501995042,5cyclictest4113233-21kworker/2:023:06:192
4059501995042,5cyclictest4104587-21kworker/2:123:46:052
4059501995042,5cyclictest4095936-21kworker/2:322:56:502
4059501995042,5cyclictest4076490-21kworker/2:222:50:572
4059501995042,5cyclictest4076490-21kworker/2:222:37:542
4059501995042,5cyclictest4064589-21kworker/2:122:17:012
4059501995042,5cyclictest4064589-21kworker/2:121:38:252
4059501995042,5cyclictest4064589-21kworker/2:121:21:392
4059501995041,5cyclictest4104587-21kworker/2:123:28:292
4059501995017,7cyclictest30-21ksoftirqd/221:41:182
4059501995015,7cyclictest30-21ksoftirqd/221:58:222
4059501995014,8cyclictest30-21ksoftirqd/223:34:002
405949699505,8cyclictest12-21ksoftirqd/000:04:250
405949699502,9cyclictest1-21systemd20:20:000
4059496995019,28cyclictest12-21ksoftirqd/021:32:520
4059496995015,5cyclictest12-21ksoftirqd/021:11:580
4059496995014,31cyclictest4063316-21modprobe20:24:260
405950499498,9cyclictest37-21ksoftirqd/319:20:023
405950499498,6cyclictest131rcu_preempt23:28:493
405950499498,12cyclictest37-21ksoftirqd/321:23:073
405950499497,5cyclictest131rcu_preempt19:55:013
405950499496,9cyclictest37-21ksoftirqd/319:12:253
405950499496,8cyclictest37-21ksoftirqd/323:18:313
4059504994939,7cyclictest37-21ksoftirqd/300:36:473
4059504994915,6cyclictest131rcu_preempt21:19:363
4059504994913,6cyclictest131rcu_preempt00:33:433
4059504994913,6cyclictest131rcu_preempt00:33:423
4059504994912,6cyclictest131rcu_preempt22:13:013
4059504994911,6cyclictest131rcu_preempt23:35:183
4059501994941,5cyclictest4104587-21kworker/2:123:11:232
4059501994941,5cyclictest4064589-21kworker/2:122:05:072
405949699497,38cyclictest4126312-21latency_hist23:30:020
4059496994941,5cyclictest12-21ksoftirqd/021:56:400
4059496994932,13cyclictest3467602-21systemd-journal19:45:010
4059496994914,27cyclictest16850irq/134-800000000:23:150
4059496994912,7cyclictest0-21swapper/022:55:470
405949699490,17cyclictest4063833-21latency_hist20:35:010
405950499489,6cyclictest131rcu_preempt00:23:103
405950499489,5cyclictest131rcu_preempt23:32:003
405950499488,5cyclictest131rcu_preempt23:52:003
405950499485,9cyclictest37-21ksoftirqd/320:00:013
4059504994814,5cyclictest131rcu_preempt22:24:263
4059504994810,6cyclictest131rcu_preempt22:56:543
405950199486,6cyclictest0-21swapper/219:50:022
4059501994840,5cyclictest4059707-21kworker/2:320:45:322
4059501994840,5cyclictest4059707-21kworker/2:320:42:102
4059501994840,5cyclictest4059707-21kworker/2:319:57:252
4059501994840,5cyclictest4059707-21kworker/2:319:41:422
4059501994840,5cyclictest4059707-21kworker/2:319:36:502
4059501994840,5cyclictest4044214-21kworker/2:119:17:252
405949899483,39cyclictest4059570-21grep19:10:011
405949699489,31cyclictest16850irq/134-800000023:47:170
405949699488,34cyclictest1-21systemd20:49:270
4059501994740,4cyclictest4059707-21kworker/2:321:02:372
4059501994739,5cyclictest4059707-21kworker/2:320:29:572
4059501994739,5cyclictest4059707-21kworker/2:320:13:182
4059501994739,5cyclictest4059707-21kworker/2:320:13:172
4059501994739,5cyclictest4059707-21kworker/2:320:07:262
4059501994639,4cyclictest4059707-21kworker/2:319:50:202
4059501994639,4cyclictest4059707-21kworker/2:319:27:312
4059501994638,5cyclictest4059707-21kworker/2:320:57:252
4059501994638,5cyclictest4059707-21kworker/2:320:50:182
4059501994638,5cyclictest4059707-21kworker/2:320:35:122
4059501994638,5cyclictest4059707-21kworker/2:319:32:032
4059501994638,5cyclictest4059707-21kworker/2:319:32:032
4059501994638,5cyclictest4059707-21kworker/2:319:24:362
405949699467,36cyclictest4059503-21latency_hist19:10:010
405949699462,14cyclictest4064463-21date20:45:020
405950499455,12cyclictest37-21ksoftirqd/320:40:003
4059504994535,7cyclictest37-21ksoftirqd/321:00:293
4059501994538,4cyclictest4059707-21kworker/2:320:21:542
4059501994538,4cyclictest4044214-21kworker/2:119:10:112
4059501994537,5cyclictest4059707-21kworker/2:321:07:552
4059501994537,5cyclictest4059707-21kworker/2:320:30:052
4059501994537,5cyclictest4059707-21kworker/2:320:04:572
405949699455,8cyclictest4060467-21latency_hist19:30:020
405950499445,9cyclictest37-21ksoftirqd/321:10:013
4059504994410,4cyclictest131rcu_preempt20:13:253
4059504994410,4cyclictest131rcu_preempt20:13:253
405949699442,9cyclictest12-21ksoftirqd/020:10:000
4059496994424,16cyclictest4061001-21sendmail-msp19:40:000
4059496994413,26cyclictest4061983-21sshd19:57:240
405950499439,6cyclictest131rcu_preempt21:00:023
405950499435,6cyclictest131rcu_preempt19:45:023
405949699437,14cyclictest4065611-21latency_hist21:05:020
405949699434,7cyclictest12-21ksoftirqd/019:45:510
405950499428,6cyclictest131rcu_preempt20:45:033
405950499425,8cyclictest37-21ksoftirqd/320:33:273
4059504994232,7cyclictest37-21ksoftirqd/319:50:013
405950499423,11cyclictest37-21ksoftirqd/320:50:003
4059496994232,5cyclictest16950irq/224-800000019:15:000
405949699422,35cyclictest0-21swapper/019:20:000
405949699421,37cyclictest0-21swapper/019:21:550
4059504994131,8cyclictest37-21ksoftirqd/320:00:433
4059504994127,8cyclictest37-21ksoftirqd/320:20:003
405949699415,9cyclictest12-21ksoftirqd/019:55:010
405949699411,11cyclictest4065234-21latency_hist21:00:020
405950499404,8cyclictest37-21ksoftirqd/320:25:003
4059504994022,10cyclictest275-21dbus-daemon19:40:003
4059496994010,25cyclictest12-21ksoftirqd/020:39:590
4059504993911,5cyclictest131rcu_preempt19:35:003
4059504993911,5cyclictest131rcu_preempt19:35:003
4059496993917,15cyclictest275-21dbus-daemon19:35:020
4059496993917,15cyclictest275-21dbus-daemon19:35:010
405950499387,5cyclictest131rcu_preempt20:28:463
4059504993712,5cyclictest131rcu_preempt19:25:013
4059496993725,9cyclictest12-21ksoftirqd/020:30:030
405950499367,4cyclictest131rcu_preempt20:51:413
405949699351,31cyclictest0-21swapper/020:50:500
405950499341,7cyclictest0-21swapper/320:06:253
405949699331,29cyclictest0-21swapper/020:00:380
4059501993221,8cyclictest4059631-21sed19:10:032
4059504992820,6cyclictest37-21ksoftirqd/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional