You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-15 - 12:55

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Tue Sep 15, 2026 00:45:24)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3695873999425,36cyclictest30-21ksoftirqd/221:15:512
3695873999419,40cyclictest3699596-21latency_hist20:20:012
3695873998919,36cyclictest3800635-21sh00:36:592
369587399872,48cyclictest3786546-21ssh00:08:012
3695873998430,29cyclictest3467602-21systemd-journal23:55:022
3695873998235,43cyclictest1-21systemd22:10:002
3695873998215,40cyclictest0-21swapper/221:51:132
3695873998215,40cyclictest0-21swapper/221:51:132
369587399782,43cyclictest3747067-21systemd-run22:45:002
3695873997824,30cyclictest3768766-21sh23:31:032
3695866997868,7cyclictest12-21ksoftirqd/021:20:000
369586699778,58cyclictest16850irq/134-800000021:55:030
369586699778,58cyclictest16850irq/134-800000021:55:020
369587399742,41cyclictest3720119-21ssh21:48:032
3695873997246,19cyclictest1-21systemd23:40:022
369587399691,40cyclictest3702084-21diskmemload23:08:502
369587399691,40cyclictest3702084-21diskmemload23:08:492
3695873996840,20cyclictest1-21systemd23:50:022
369586699689,5cyclictest131rcu_preempt21:15:020
3695873996728,35cyclictest3736797-21sh22:23:172
3695873996727,36cyclictest3748107-21sh22:46:592
3695873996719,43cyclictest3700291-21sed20:35:012
369587399671,62cyclictest3712923-21ssh21:32:512
3695866996732,15cyclictest131rcu_preempt23:43:140
3695873996620,42cyclictest30-21ksoftirqd/223:42:332
369587399661,61cyclictest3764506-21ssh23:22:072
369587399661,60cyclictest3731048-21ssh22:10:542
369587399652,59cyclictest3783450-21ssh00:01:202
369587399652,59cyclictest3699712-21munin-run20:25:002
3695873996523,38cyclictest3760958-21latency_hist23:15:012
369587399651,60cyclictest3795893-21ssh00:27:142
369587399651,60cyclictest3779995-21ssh23:54:332
369587399651,60cyclictest3750960-21ssh22:53:122
369587399651,60cyclictest3742581-21ssh22:35:122
369587399651,60cyclictest3725452-21ssh21:59:242
369587399651,38cyclictest3738993-21ssh22:27:522
369586699652,6cyclictest131rcu_preempt00:20:030
369586699652,47cyclictest151rcuc/021:26:290
3695866996514,6cyclictest131rcu_preempt00:36:540
369587399649,3cyclictest131rcu_preempt19:30:022
3695873996419,39cyclictest3697347-21modprobe19:40:002
369587399641,59cyclictest3798885-21ssh00:33:322
369587399641,59cyclictest3793130-21ssh00:21:192
369587399641,59cyclictest3791897-21ssh00:19:082
369587399641,59cyclictest3761697-21ssh23:16:082
369587399641,59cyclictest3754796-21ssh23:01:052
369587399641,59cyclictest3740506-21ssh22:30:452
369587399641,58cyclictest3752325-21ssh22:55:532
369587399641,58cyclictest3716750-21ssh21:40:422
369587399641,58cyclictest3716750-21ssh21:40:412
369587399641,58cyclictest3710046-21ssh21:26:382
3695866996448,13cyclictest16950irq/224-800000020:05:020
369587399632,57cyclictest3708825-21ssh21:24:132
369587399632,56cyclictest3696612-21munin-run19:25:002
369587399631,58cyclictest3787848-21ssh00:10:312
369587399631,58cyclictest3767271-21ssh23:28:052
369587399631,58cyclictest3733473-21ssh22:15:532
369587399631,58cyclictest3726877-21ssh22:02:102
369587399631,58cyclictest3726877-21ssh22:02:092
369587399631,58cyclictest3714561-21ssh21:36:072
369587399631,58cyclictest3702949-21ssh21:11:342
3695866996324,10cyclictest12-21ksoftirqd/022:40:010
369587399622,54cyclictest3701251-21munin-run20:55:002
369587399621,55cyclictest3698920-21munin-run20:10:002
369587399612,54cyclictest3698429-21munin-run20:00:012
3695873996123,34cyclictest3696054-21latency_hist19:15:032
369587399611,56cyclictest3699963-21munin-run20:30:012
369587399611,56cyclictest3696301-21sshd19:19:312
3695866996118,13cyclictest12-21ksoftirqd/020:20:010
3695876996013,5cyclictest131rcu_preempt23:35:053
369587399602,53cyclictest3697897-21munin-run19:50:012
369587399601,55cyclictest3698107-21sshd19:52:032
369587399601,55cyclictest3697596-21sshd19:40:312
369586699609,6cyclictest131rcu_preempt22:05:010
369586699609,6cyclictest131rcu_preempt22:05:010
369586699603,8cyclictest12-21ksoftirqd/022:31:330
369587399592,53cyclictest3698679-21munin-run20:05:022
369587399592,51cyclictest3699184-21munin-run20:15:012
369587399591,54cyclictest3701483-21sshd20:59:312
369587399591,54cyclictest3701483-21sshd20:59:312
369587399591,54cyclictest364-21snmpd21:00:042
369586699598,6cyclictest131rcu_preempt23:50:020
369587699589,6cyclictest131rcu_preempt00:00:403
369587699583,13cyclictest37-21ksoftirqd/322:37:373
3695876995812,6cyclictest131rcu_preempt00:25:023
3695876995811,5cyclictest131rcu_preempt23:57:133
369587399582,52cyclictest3700998-21munin-run20:50:002
369587399582,52cyclictest3700476-21munin-run20:40:002
369587399581,52cyclictest3702090-21munin-run21:10:002
3695866995837,18cyclictest12-21ksoftirqd/022:09:230
369586699581,39cyclictest3708625-21ssh21:23:470
369587699578,6cyclictest131rcu_preempt23:35:003
3695876995714,5cyclictest131rcu_preempt21:55:023
3695876995712,6cyclictest131rcu_preempt00:39:243
3695876995710,6cyclictest131rcu_preempt22:12:503
3695876995710,6cyclictest131rcu_preempt21:51:273
3695876995710,6cyclictest131rcu_preempt21:51:273
369586699576,9cyclictest12-21ksoftirqd/022:50:010
3695866995730,11cyclictest3759675-21ssh23:12:010
369587699569,6cyclictest131rcu_preempt22:40:213
369587699568,6cyclictest131rcu_preempt23:55:013
3695876995615,6cyclictest131rcu_preempt22:58:113
3695876995615,5cyclictest131rcu_preempt00:11:333
3695876995614,6cyclictest131rcu_preempt21:00:003
3695876995614,6cyclictest131rcu_preempt21:00:003
3695876995614,5cyclictest131rcu_preempt23:14:353
3695876995612,7cyclictest131rcu_preempt22:47:303
369587399563,49cyclictest364-21snmpd19:32:222
369587399563,49cyclictest364-21snmpd19:32:222
3695871995619,25cyclictest23-21ksoftirqd/122:36:311
369586699563,6cyclictest275-21dbus-daemon23:05:000
3695866995627,14cyclictest3763910-21ssh23:20:450
3695866995622,8cyclictest3797379-21sh00:30:060
3695866995613,9cyclictest284-21systemd-logind20:00:010
369586699561,38cyclictest3761762-21ssh23:16:160
3695866995610,6cyclictest131rcu_preempt22:59:350
3695876995513,6cyclictest131rcu_preempt21:14:183
3695876995510,5cyclictest131rcu_preempt21:25:063
369587399553,48cyclictest3700916-21cat20:45:022
3695871995537,14cyclictest1-21systemd23:20:011
369586699556,31cyclictest12-21ksoftirqd/000:25:240
3695866995527,14cyclictest3746005-21ssh22:42:320
3695866995519,6cyclictest16850irq/134-800000021:34:320
3695866995519,32cyclictest16850irq/134-800000023:09:150
3695866995519,32cyclictest16850irq/134-800000023:09:150
3695866995518,32cyclictest16850irq/134-800000023:29:040
369587699549,13cyclictest37-21ksoftirqd/300:33:333
369587699548,12cyclictest37-21ksoftirqd/321:20:553
3695876995415,9cyclictest37-21ksoftirqd/323:05:293
3695876995415,9cyclictest37-21ksoftirqd/323:05:283
3695876995413,5cyclictest131rcu_preempt23:50:003
3695876995413,5cyclictest131rcu_preempt21:15:583
3695876995411,5cyclictest131rcu_preempt21:34:323
3695876995410,6cyclictest131rcu_preempt21:37:323
369587199542,7cyclictest23-21ksoftirqd/121:47:511
3695871995426,14cyclictest23-21ksoftirqd/122:00:001
369586699542,42cyclictest3700460-21modprobe20:40:000
3695866995421,9cyclictest12-21ksoftirqd/000:02:410
3695866995419,5cyclictest16850irq/134-800000000:06:090
3695866995418,31cyclictest16850irq/134-800000023:58:330
3695866995414,36cyclictest16850irq/134-800000022:28:340
3695866995410,6cyclictest131rcu_preempt23:33:500
369587699539,8cyclictest37-21ksoftirqd/323:25:023
369587699539,5cyclictest131rcu_preempt22:20:193
369587699539,5cyclictest131rcu_preempt00:16:323
369587699537,6cyclictest131rcu_preempt00:27:133
3695876995316,6cyclictest131rcu_preempt22:10:013
3695876995314,6cyclictest131rcu_preempt22:25:363
3695876995313,6cyclictest131rcu_preempt23:26:523
3695871995310,6cyclictest131rcu_preempt23:45:031
3695866995325,13cyclictest3724096-21ssh21:56:140
3695866995318,31cyclictest16850irq/134-800000022:54:160
3695866995318,31cyclictest16850irq/134-800000021:41:190
3695866995318,31cyclictest16850irq/134-800000021:41:180
3695866995317,32cyclictest16850irq/134-800000023:51:200
369586699531,37cyclictest3770808-21ssh23:35:220
369587699528,5cyclictest131rcu_preempt23:18:413
369587699526,9cyclictest37-21ksoftirqd/323:42:223
369587699524,10cyclictest37-21ksoftirqd/321:49:093
3695876995213,6cyclictest131rcu_preempt23:04:103
3695876995211,5cyclictest131rcu_preempt22:51:443
3695876995210,5cyclictest131rcu_preempt19:15:013
369587199521,7cyclictest0-21swapper/100:02:491
3695866995218,17cyclictest16850irq/134-800000022:12:350
3695866995217,32cyclictest16850irq/134-800000022:21:570
3695866995215,19cyclictest131rcu_preempt20:55:010
369586699521,11cyclictest12-21ksoftirqd/019:49:110
369587699518,6cyclictest131rcu_preempt00:05:033
369587699516,8cyclictest37-21ksoftirqd/322:00:033
369587699516,8cyclictest37-21ksoftirqd/322:00:023
3695876995118,6cyclictest131rcu_preempt22:31:373
3695876995111,5cyclictest131rcu_preempt19:17:013
3695876995110,6cyclictest131rcu_preempt22:18:383
3695876995110,5cyclictest131rcu_preempt19:30:003
369586699516,17cyclictest131rcu_preempt00:10:330
3695866995132,15cyclictest131rcu_preempt19:35:000
3695866995132,15cyclictest131rcu_preempt19:35:000
3695866995115,32cyclictest16850irq/134-800000021:35:030
3695876995013,6cyclictest131rcu_preempt21:40:243
3695876995013,6cyclictest131rcu_preempt21:40:243
369587199501,19cyclictest0-21swapper/121:35:081
3695866995018,16cyclictest16850irq/134-800000022:17:250
3695866995017,5cyclictest16850irq/134-800000000:20:290
3695866995015,6cyclictest16850irq/134-800000021:46:280
3695876994911,5cyclictest131rcu_preempt19:22:003
369587199496,7cyclictest23-21ksoftirqd/123:00:001
3695871994931,13cyclictest3467602-21systemd-journal00:35:321
369586699491,8cyclictest12-21ksoftirqd/019:14:160
369587699486,9cyclictest37-21ksoftirqd/320:37:123
3695876994813,5cyclictest131rcu_preempt19:49:313
369587199487,6cyclictest131rcu_preempt20:20:001
3695871994831,7cyclictest23-21ksoftirqd/119:20:001
369586699483,5cyclictest131rcu_preempt19:45:000
369586699481,35cyclictest12-21ksoftirqd/019:20:350
369587199476,5cyclictest131rcu_preempt22:45:001
369587199471,5cyclictest0-21swapper/123:39:191
3695871994714,20cyclictest23-21ksoftirqd/121:22:241
3695871994714,15cyclictest23-21ksoftirqd/123:30:451
3695871994711,6cyclictest131rcu_preempt00:20:001
3695876994611,6cyclictest131rcu_preempt20:45:033
3695871994624,15cyclictest3467602-21systemd-journal20:05:001
3695871994615,21cyclictest0-21swapper/123:52:301
369586699462,6cyclictest131rcu_preempt19:30:020
3695866994610,6cyclictest131rcu_preempt20:50:020
3695876994519,6cyclictest361rcuc/320:30:023
3695876994511,6cyclictest131rcu_preempt19:40:013
3695871994535,7cyclictest23-21ksoftirqd/120:28:471
3695871994526,13cyclictest275-21dbus-daemon00:00:011
3695871994524,14cyclictest23-21ksoftirqd/119:15:011
3695871994521,9cyclictest131rcu_preempt00:06:241
3695871994520,21cyclictest23-21ksoftirqd/122:20:101
369587199451,9cyclictest0-21swapper/122:18:231
369587199451,9cyclictest0-21swapper/100:22:411
369587199451,8cyclictest0-21swapper/122:07:081
3695871994515,20cyclictest23-21ksoftirqd/123:22:301
369587199451,18cyclictest0-21swapper/100:31:541
3695866994527,14cyclictest275-21dbus-daemon20:45:000
369587699446,8cyclictest37-21ksoftirqd/319:42:033
3695876994429,12cyclictest1-21systemd20:10:013
369587699441,8cyclictest37-21ksoftirqd/320:00:023
369587199448,6cyclictest131rcu_preempt21:10:001
3695871994428,10cyclictest0-21swapper/122:05:011
3695871994428,10cyclictest0-21swapper/122:05:011
3695871994421,18cyclictest23-21ksoftirqd/122:52:451
3695871994420,15cyclictest23-21ksoftirqd/121:40:391
3695871994420,15cyclictest23-21ksoftirqd/121:40:381
369587199441,8cyclictest0-21swapper/121:15:291
3695871994414,24cyclictest221rcuc/120:25:011
369586699443,5cyclictest0-21swapper/020:30:020
3695866994428,11cyclictest1-21systemd19:20:010
3695866994411,4cyclictest131rcu_preempt21:05:000
369587699431,5cyclictest0-21swapper/320:17:123
369587399431,38cyclictest3695900-21latency_hist19:10:012
369587399431,38cyclictest3695900-21latency_hist19:10:012
369587199438,5cyclictest131rcu_preempt21:34:161
369587199437,6cyclictest131rcu_preempt21:26:561
3695871994334,6cyclictest221rcuc/120:10:011
3695871994327,11cyclictest3789885-21munin-run00:15:011
369587199432,5cyclictest131rcu_preempt23:30:001
369587199431,7cyclictest0-21swapper/121:11:261
3695871994317,9cyclictest131rcu_preempt22:14:411
369587199431,6cyclictest131rcu_preempt19:10:011
369587199431,6cyclictest131rcu_preempt19:10:011
3695871994310,6cyclictest131rcu_preempt22:45:171
369586699437,31cyclictest0-21swapper/021:10:010
3695866994335,4cyclictest16950irq/224-800000021:00:010
3695866994335,4cyclictest16950irq/224-800000021:00:000
3695866994311,5cyclictest131rcu_preempt20:25:010
369587699427,5cyclictest131rcu_preempt20:32:113
369587699426,6cyclictest131rcu_preempt21:05:023
3695876994228,11cyclictest3467602-21systemd-journal19:32:323
3695876994228,11cyclictest3467602-21systemd-journal19:32:313
369587199427,6cyclictest131rcu_preempt23:48:571
369587199424,8cyclictest23-21ksoftirqd/120:38:321
3695871994221,14cyclictest3467602-21systemd-journal20:15:001
369587199421,6cyclictest0-21swapper/122:29:131
369587199421,5cyclictest131rcu_preempt21:52:381
369587199421,5cyclictest131rcu_preempt21:52:381
3695866994225,13cyclictest1-21systemd20:10:000
369587199419,6cyclictest131rcu_preempt19:40:001
3695871994124,10cyclictest23-21ksoftirqd/121:00:011
3695871994124,10cyclictest23-21ksoftirqd/121:00:001
3695871994117,10cyclictest23-21ksoftirqd/123:15:001
3695871994116,9cyclictest131rcu_preempt23:09:231
3695871994116,9cyclictest131rcu_preempt23:09:231
369587199411,16cyclictest0-21swapper/123:00:221
369586699418,7cyclictest12-21ksoftirqd/019:37:030
3695866994127,10cyclictest309-21in:imuxsock20:32:310
369587699401,8cyclictest37-21ksoftirqd/320:45:023
3695871994025,8cyclictest3467602-21systemd-journal19:30:001
369586699409,5cyclictest131rcu_preempt19:55:000
3695871993918,16cyclictest131rcu_preempt00:27:571
3695871993914,12cyclictest23-21ksoftirqd/122:30:161
3695866993920,8cyclictest12-21ksoftirqd/020:14:010
369587699384,6cyclictest131rcu_preempt20:55:013
3695876993822,13cyclictest37-21ksoftirqd/320:05:013
369587699381,6cyclictest150-21pr/ttyS219:51:483
3695871993826,9cyclictest23-21ksoftirqd/120:55:011
369587699376,5cyclictest131rcu_preempt21:01:353
369587199376,6cyclictest131rcu_preempt20:35:001
3695871993725,9cyclictest23-21ksoftirqd/119:32:491
3695871993725,9cyclictest23-21ksoftirqd/119:32:491
3695871993722,10cyclictest23-21ksoftirqd/119:51:311
369587699361,6cyclictest150-21pr/ttyS220:14:463
3695876993612,6cyclictest131rcu_preempt19:10:023
3695876993612,6cyclictest131rcu_preempt19:10:013
3695871993623,9cyclictest23-21ksoftirqd/119:25:011
3695876993524,9cyclictest37-21ksoftirqd/320:25:023
3695871993520,11cyclictest3467602-21systemd-journal19:59:031
3695866993525,7cyclictest12-21ksoftirqd/019:10:010
3695866993525,7cyclictest12-21ksoftirqd/019:10:010
369587199333,5cyclictest131rcu_preempt21:00:311
369587199332,4cyclictest131rcu_preempt20:43:391
3695871993322,8cyclictest3467602-21systemd-journal20:47:311
369587199331,5cyclictest131rcu_preempt19:45:021
369587199321,8cyclictest131rcu_preempt19:46:281
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional