You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-27 - 18:46

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Mon Apr 27, 2026 12:45:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3452410998415,39cyclictest0-21swapper/009:15:030
3452410998324,40cyclictest150-21pr/ttyS212:15:210
3452410998112,49cyclictest0-21swapper/011:40:110
3452410997911,38cyclictest0-21swapper/012:15:000
3452410997843,4cyclictest3453978-21munin-run07:40:010
3452410997832,20cyclictest0-21swapper/010:36:050
3452410997826,14cyclictest3494349-21ssh10:33:020
3452410997740,34cyclictest3475821-21/usr/sbin/munin09:50:010
3452410997641,4cyclictest3490895-21munin-run10:25:010
3452410997626,19cyclictest3487114-21sh10:15:540
3452410997624,20cyclictest3454334-21cat07:45:020
3452410997624,20cyclictest3454334-21cat07:45:020
3452410997529,13cyclictest3468028-21sh09:31:330
3452410997523,14cyclictest3513950-21ssh11:18:510
3452410997422,15cyclictest3454814-21latency_hist07:55:020
3452410997413,34cyclictest0-21swapper/012:34:150
3452410997324,14cyclictest3455248-21sh08:05:000
3452410997229,13cyclictest3506283-21sh11:00:300
3452419997135,25cyclictest30-21ksoftirqd/208:45:022
3452410997126,31cyclictest0-21swapper/012:28:030
3452410997124,43cyclictest3480160-21ssh10:00:000
3452410997122,15cyclictest3457242-21cat08:40:020
3452410997121,15cyclictest3502693-21ssh10:52:270
3452410997023,14cyclictest3532947-21ssh12:03:030
3452410997010,29cyclictest0-21swapper/009:16:420
3452410996920,14cyclictest3521500-21ssh11:36:100
3452410996829,35cyclictest3467602-21systemd-journal12:10:000
3452410996829,35cyclictest3467602-21systemd-journal12:10:000
345242199676,6cyclictest37-21ksoftirqd/309:55:013
345241999677,9cyclictest30-21ksoftirqd/211:45:002
3452410996733,29cyclictest3473745-21latency_hist09:45:020
3452410996728,10cyclictest3547920-21ssh12:37:420
3452410996722,12cyclictest3458170-21latency_hist09:00:020
3452410996721,16cyclictest3454580-21latency_hist07:50:010
345241599666,43cyclictest3482449-21ssh10:05:021
345241599666,43cyclictest3482449-21ssh10:05:021
3452410996625,12cyclictest3508365-21ssh11:05:250
3452410996610,23cyclictest3491301-21rm10:25:380
3452410996533,28cyclictest150-21pr/ttyS211:29:590
3452410996527,13cyclictest3486547-21munin-run10:15:010
3452410996524,36cyclictest0-21swapper/009:29:020
3452410996516,17cyclictest0-21swapper/011:55:550
345241099633,10cyclictest131rcu_preempt10:45:020
3452410996322,36cyclictest0-21swapper/010:05:020
3452410996322,36cyclictest0-21swapper/010:05:020
345241999626,6cyclictest30-21ksoftirqd/212:25:012
3452410996226,5cyclictest3484387-21latency_hist10:10:010
3452410996226,32cyclictest0-21swapper/007:15:440
3452410996226,32cyclictest0-21swapper/007:15:430
3452410996222,14cyclictest3504557-21ssh10:56:390
3452410996222,10cyclictest3464743-21ssh09:24:070
3452410996219,11cyclictest3525654-21ssh11:45:460
3452410996219,11cyclictest3525654-21ssh11:45:460
345241999614,10cyclictest30-21ksoftirqd/210:05:012
345241999614,10cyclictest30-21ksoftirqd/210:05:012
3452410996119,38cyclictest0-21swapper/008:00:020
3452410996119,38cyclictest0-21swapper/008:00:010
3452421996021,10cyclictest37-21ksoftirqd/309:34:083
345241099609,9cyclictest3458618-21diskmemload09:35:110
3452410996047,10cyclictest3457293-21sshd08:40:260
3452410996043,13cyclictest3458641-21munin-run09:10:010
3452410996042,14cyclictest3457815-21sshd08:54:440
3452410996041,13cyclictest3456000-21cron08:20:000
3452410996022,11cyclictest0-21swapper/011:51:530
3452421995920,9cyclictest131rcu_preempt10:21:133
3452421995914,8cyclictest37-21ksoftirqd/312:38:043
345241999596,11cyclictest30-21ksoftirqd/210:20:012
3452410995942,13cyclictest150-21pr/ttyS211:10:370
345241099594,20cyclictest3478054-21latency_hist09:55:020
3452410995919,12cyclictest3541111-21ssh12:21:470
3452410995917,38cyclictest0-21swapper/007:10:020
3452410995917,20cyclictest0-21swapper/011:32:360
345241999587,8cyclictest30-21ksoftirqd/209:33:372
345241999586,9cyclictest30-21ksoftirqd/211:35:482
345241999584,7cyclictest30-21ksoftirqd/210:00:002
3452419995818,11cyclictest30-21ksoftirqd/209:10:012
3452419995816,7cyclictest30-21ksoftirqd/210:41:382
3452410995845,10cyclictest3456294-21munin-run08:25:000
3452410995842,13cyclictest3456811-21munin-run08:35:010
3452410995842,13cyclictest3456811-21munin-run08:35:010
345241999576,11cyclictest30-21ksoftirqd/210:56:012
345241999575,8cyclictest30-21ksoftirqd/211:47:032
345241999575,8cyclictest30-21ksoftirqd/211:47:032
345241999575,8cyclictest30-21ksoftirqd/209:19:142
3452410995737,17cyclictest364-21snmpd10:45:050
345242199561,19cyclictest37-21ksoftirqd/310:50:203
3452421995610,8cyclictest37-21ksoftirqd/312:30:023
345241999566,8cyclictest30-21ksoftirqd/210:10:202
345241999564,8cyclictest30-21ksoftirqd/211:56:292
345241999564,11cyclictest30-21ksoftirqd/211:29:262
345241999562,10cyclictest30-21ksoftirqd/212:15:012
345241999561,17cyclictest30-21ksoftirqd/209:42:092
3452410995644,9cyclictest3457551-21sshd08:46:440
3452410995642,10cyclictest3458375-21munin-run09:05:000
3452410995615,12cyclictest3458618-21diskmemload11:20:090
345242199559,8cyclictest37-21ksoftirqd/311:45:053
345242199559,8cyclictest37-21ksoftirqd/311:45:053
345242199558,10cyclictest37-21ksoftirqd/310:16:543
345242199555,10cyclictest37-21ksoftirqd/311:45:013
3452421995517,10cyclictest37-21ksoftirqd/312:10:563
345241999557,5cyclictest131rcu_preempt10:51:092
345241999555,10cyclictest30-21ksoftirqd/210:09:212
3452419995546,5cyclictest3454528-21kworker/2:108:20:012
3452419995543,9cyclictest30-21ksoftirqd/212:00:142
345241599556,7cyclictest48-21kcompactd012:34:171
3452410995542,9cyclictest3455754-21munin-run08:15:010
3452410995542,9cyclictest3455754-21munin-run08:15:010
3452410995525,28cyclictest0-21swapper/007:12:440
345242199547,8cyclictest37-21ksoftirqd/308:35:003
345242199547,8cyclictest37-21ksoftirqd/308:35:003
345242199547,12cyclictest37-21ksoftirqd/311:23:193
345242199546,9cyclictest37-21ksoftirqd/312:09:353
345242199546,9cyclictest37-21ksoftirqd/312:09:353
345242199546,13cyclictest37-21ksoftirqd/311:17:523
345242199542,12cyclictest361rcuc/310:00:533
345242199542,12cyclictest361rcuc/310:00:533
3452421995410,10cyclictest37-21ksoftirqd/309:16:003
345241999547,9cyclictest30-21ksoftirqd/209:36:442
345241999546,8cyclictest30-21ksoftirqd/207:55:012
345241999544,8cyclictest30-21ksoftirqd/211:12:442
345241999543,7cyclictest30-21ksoftirqd/209:11:342
3452419995411,5cyclictest131rcu_preempt09:52:212
3452415995439,11cyclictest3480108-21cron10:00:001
3452415995436,14cyclictest3467602-21systemd-journal11:19:591
345241599541,13cyclictest23-21ksoftirqd/111:27:171
3452410995438,12cyclictest3456515-21sshd08:26:450
3452410995420,30cyclictest3453329-21latency_hist07:25:020
345242199538,8cyclictest37-21ksoftirqd/309:10:093
345242199534,31cyclictest37-21ksoftirqd/308:25:023
345241999537,10cyclictest30-21ksoftirqd/210:21:032
345241999535,11cyclictest30-21ksoftirqd/212:32:332
345241999534,8cyclictest30-21ksoftirqd/209:21:502
345241999534,7cyclictest30-21ksoftirqd/211:05:022
3452419995319,11cyclictest30-21ksoftirqd/207:32:442
3452419995319,11cyclictest30-21ksoftirqd/207:32:442
3452419995310,6cyclictest131rcu_preempt12:38:512
3452410995341,9cyclictest364-21snmpd08:08:300
345242199526,9cyclictest37-21ksoftirqd/309:39:403
345242199526,6cyclictest37-21ksoftirqd/310:26:453
345242199525,8cyclictest37-21ksoftirqd/310:33:043
345242199525,8cyclictest37-21ksoftirqd/309:43:503
345242199525,10cyclictest37-21ksoftirqd/311:10:253
345242199524,11cyclictest37-21ksoftirqd/309:55:443
3452421995237,11cyclictest1-21systemd08:40:003
345242199522,6cyclictest37-21ksoftirqd/307:50:003
3452421995217,8cyclictest37-21ksoftirqd/310:59:343
3452421995215,7cyclictest37-21ksoftirqd/312:20:183
3452421995214,8cyclictest131rcu_preempt10:40:003
3452421995213,7cyclictest37-21ksoftirqd/312:32:203
3452421995212,8cyclictest37-21ksoftirqd/311:33:103
345241999527,11cyclictest30-21ksoftirqd/212:16:182
345241999526,8cyclictest30-21ksoftirqd/211:30:092
345241999526,12cyclictest30-21ksoftirqd/209:28:242
345241999524,8cyclictest30-21ksoftirqd/209:45:362
345241999523,9cyclictest30-21ksoftirqd/211:09:592
3452419995210,5cyclictest131rcu_preempt10:49:472
3452419995210,10cyclictest30-21ksoftirqd/211:16:512
345242199516,6cyclictest37-21ksoftirqd/311:04:383
345242199514,8cyclictest37-21ksoftirqd/311:51:233
345242199514,10cyclictest37-21ksoftirqd/310:12:183
345241999516,9cyclictest30-21ksoftirqd/211:23:352
345241999516,7cyclictest30-21ksoftirqd/212:27:202
345241999514,7cyclictest30-21ksoftirqd/212:09:392
345241999514,7cyclictest30-21ksoftirqd/212:09:392
345241999514,7cyclictest30-21ksoftirqd/208:55:012
345241999513,15cyclictest30-21ksoftirqd/210:36:532
3452419995111,8cyclictest30-21ksoftirqd/210:26:062
345241599519,4cyclictest131rcu_preempt11:07:441
3452415995134,12cyclictest3467602-21systemd-journal08:50:011
345242199505,8cyclictest37-21ksoftirqd/311:26:103
345242199504,9cyclictest37-21ksoftirqd/312:17:093
345242199504,8cyclictest37-21ksoftirqd/309:27:353
345241999506,11cyclictest30-21ksoftirqd/210:32:142
3452419995034,12cyclictest1-21systemd08:10:012
345241599505,4cyclictest131rcu_preempt10:19:021
3452415995040,6cyclictest23-21ksoftirqd/111:40:011
3452415995013,7cyclictest131rcu_preempt09:20:001
3452410995038,8cyclictest364-21snmpd07:27:590
3452410995037,9cyclictest48-21kcompactd007:33:480
3452410995037,9cyclictest48-21kcompactd007:33:480
345242199496,7cyclictest37-21ksoftirqd/309:45:493
345242199495,5cyclictest37-21ksoftirqd/310:46:153
3452421994917,7cyclictest37-21ksoftirqd/310:06:563
3452421994914,8cyclictest37-21ksoftirqd/309:20:153
3452421994913,6cyclictest131rcu_preempt12:01:073
345242199491,14cyclictest3458745-21sed09:10:023
345241999499,5cyclictest131rcu_preempt11:50:052
345241999497,10cyclictest30-21ksoftirqd/208:25:022
345241999496,13cyclictest30-21ksoftirqd/209:05:002
345241999495,9cyclictest30-21ksoftirqd/207:29:592
3452419994942,4cyclictest3444192-21kworker/2:207:14:512
3452419994942,4cyclictest3429555-21kworker/2:307:43:252
3452419994942,4cyclictest3429555-21kworker/2:307:43:252
3452419994941,5cyclictest3454528-21kworker/2:108:33:442
3452419994941,5cyclictest3454528-21kworker/2:108:33:442
345241999492,14cyclictest30-21ksoftirqd/208:00:012
345241999492,14cyclictest30-21ksoftirqd/208:00:012
3452419994910,5cyclictest131rcu_preempt07:20:022
3452419994910,5cyclictest131rcu_preempt07:20:022
345241599497,5cyclictest0-21swapper/110:27:111
345241599493,9cyclictest23-21ksoftirqd/110:48:381
345241599491,8cyclictest0-21swapper/110:10:001
3452415994917,5cyclictest131rcu_preempt09:27:141
345242199487,9cyclictest0-21swapper/311:40:013
345242199485,9cyclictest37-21ksoftirqd/310:42:123
345242199485,8cyclictest37-21ksoftirqd/311:09:353
345242199483,9cyclictest37-21ksoftirqd/311:55:333
345241999484,6cyclictest30-21ksoftirqd/208:38:442
3452419994841,4cyclictest3444192-21kworker/2:208:29:332
3452419994840,5cyclictest3454528-21kworker/2:108:59:222
3452419994840,5cyclictest3454528-21kworker/2:108:49:002
3452419994840,5cyclictest3429555-21kworker/2:307:22:512
3452419994839,5cyclictest3429555-21kworker/2:307:47:162
345241599484,6cyclictest131rcu_preempt12:18:531
3452419994740,4cyclictest3429555-21kworker/2:307:36:222
3452419994739,5cyclictest3454528-21kworker/2:108:10:432
3452419994739,5cyclictest3454528-21kworker/2:108:10:432
3452419994739,5cyclictest3454528-21kworker/2:108:00:312
345241599478,12cyclictest23-21ksoftirqd/112:03:181
345241599478,10cyclictest23-21ksoftirqd/110:45:001
345241599478,10cyclictest23-21ksoftirqd/109:25:031
345241599473,6cyclictest131rcu_preempt08:00:021
345241599473,6cyclictest131rcu_preempt08:00:021
3452415994733,10cyclictest1-21systemd08:40:001
345241599471,7cyclictest0-21swapper/109:11:001
345241599471,6cyclictest0-21swapper/112:20:111
3452415994713,8cyclictest23-21ksoftirqd/112:35:111
345241599464,7cyclictest23-21ksoftirqd/111:42:381
3452415994638,5cyclictest23-21ksoftirqd/110:12:451
3452415994637,6cyclictest23-21ksoftirqd/109:10:001
3452415994635,9cyclictest23-21ksoftirqd/112:07:381
3452415994635,9cyclictest23-21ksoftirqd/112:07:371
3452415994623,19cyclictest23-21ksoftirqd/111:32:041
3452415994618,16cyclictest23-21ksoftirqd/108:40:441
3452415994610,6cyclictest131rcu_preempt10:53:381
345242199458,7cyclictest3453283-21ls07:25:013
3452421994532,11cyclictest37-21ksoftirqd/308:15:013
3452421994532,11cyclictest37-21ksoftirqd/308:15:013
345241599459,4cyclictest131rcu_preempt10:56:161
345241599458,6cyclictest131rcu_preempt11:45:181
345241599458,6cyclictest131rcu_preempt11:45:181
345241599456,8cyclictest23-21ksoftirqd/110:33:301
345241599456,12cyclictest23-21ksoftirqd/107:30:001
345241599453,6cyclictest23-21ksoftirqd/111:54:561
345241599453,5cyclictest23-21ksoftirqd/109:39:261
3452415994532,9cyclictest23-21ksoftirqd/112:14:471
345241599453,10cyclictest23-21ksoftirqd/111:24:381
3452415994529,9cyclictest23-21ksoftirqd/111:01:461
3452415994525,10cyclictest23-21ksoftirqd/110:40:021
3452415994518,18cyclictest23-21ksoftirqd/112:28:361
345241599451,6cyclictest0-21swapper/107:50:441
3452415994515,13cyclictest23-21ksoftirqd/107:20:011
3452415994515,13cyclictest23-21ksoftirqd/107:20:011
3452415994513,8cyclictest131rcu_preempt09:50:331
3452415994512,8cyclictest131rcu_preempt10:22:131
3452415994511,4cyclictest131rcu_preempt11:13:121
345242199446,7cyclictest37-21ksoftirqd/309:00:013
3452421994426,14cyclictest3467602-21systemd-journal08:10:013
3452421994410,8cyclictest0-21swapper/307:58:293
3452421994410,8cyclictest0-21swapper/307:58:293
3452415994432,8cyclictest1-21systemd08:15:011
3452415994432,8cyclictest1-21systemd08:15:011
3452415994419,16cyclictest23-21ksoftirqd/109:31:171
345242199434,9cyclictest37-21ksoftirqd/308:50:023
345241599434,7cyclictest131rcu_preempt09:41:471
3452415994329,11cyclictest275-21dbus-daemon09:05:001
3452415994326,8cyclictest23-21ksoftirqd/111:55:151
345241599431,5cyclictest0-21swapper/108:52:441
345242199429,7cyclictest0-21swapper/307:53:383
345242199425,10cyclictest37-21ksoftirqd/307:40:013
3452421994224,13cyclictest1-21systemd08:05:003
345242199421,7cyclictest0-21swapper/308:25:233
3452415994231,8cyclictest23-21ksoftirqd/108:35:011
3452415994231,8cyclictest23-21ksoftirqd/108:35:011
345241599421,8cyclictest0-21swapper/108:20:551
345241599421,5cyclictest131rcu_preempt09:49:191
345242199417,5cyclictest131rcu_preempt07:30:053
345242199417,5cyclictest131rcu_preempt07:30:053
345242199412,9cyclictest37-21ksoftirqd/307:15:023
3452421994129,8cyclictest3467602-21systemd-journal08:50:033
3452421994128,9cyclictest3467602-21systemd-journal08:20:003
3452415994126,11cyclictest3467602-21systemd-journal07:39:321
345242199406,7cyclictest37-21ksoftirqd/307:10:023
345242199405,9cyclictest37-21ksoftirqd/307:45:013
345242199405,9cyclictest37-21ksoftirqd/307:45:013
3452421994025,11cyclictest3467602-21systemd-journal08:41:443
345241599406,8cyclictest23-21ksoftirqd/107:22:131
3452415994031,6cyclictest23-21ksoftirqd/107:45:011
3452415994031,6cyclictest23-21ksoftirqd/107:45:011
3452415994022,13cyclictest3467602-21systemd-journal07:35:011
3452415994022,13cyclictest3467602-21systemd-journal07:35:011
3452415994012,8cyclictest131rcu_preempt07:12:131
3452415994012,6cyclictest131rcu_preempt08:19:581
345242199397,4cyclictest131rcu_preempt07:25:043
345242199394,8cyclictest0-21swapper/307:20:013
345242199394,8cyclictest0-21swapper/307:20:013
3452421993930,5cyclictest3458382-21latency_hist09:05:013
345241599395,9cyclictest23-21ksoftirqd/107:46:441
345241599392,9cyclictest23-21ksoftirqd/108:25:211
345241599392,24cyclictest3454701-21kworker/u8:008:05:021
345241599392,24cyclictest0-21swapper/107:10:021
345241599381,6cyclictest0-21swapper/108:57:261
345241599351,8cyclictest0-21swapper/108:01:441
3452419993224,6cyclictest30-21ksoftirqd/207:10:012
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional