You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-10 - 21:39

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Mon Aug 10, 2026 12:45:10)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1187799564,15cyclictest21-21ksoftirqd/109:10:121
1188599541,46cyclictest22933-21apt-get07:55:122
1188599532,36cyclictest32571-21apt-get10:05:132
1187799513,8cyclictest21-21ksoftirqd/109:40:111
1188599491,38cyclictest0-21swapper/209:50:132
1188599491,36cyclictest32695-21apt-get11:35:142
1187799491,8cyclictest0-21swapper/107:15:121
1188599461,33cyclictest0-21swapper/211:10:182
1186699464,11cyclictest131rcu_preempt11:25:130
1189399451,23cyclictest35966irq/58-eth010:20:123
1189399451,23cyclictest35966irq/58-eth010:20:123
1188599451,39cyclictest0-21swapper/211:20:142
1187799456,30cyclictest29074-21ssh11:15:131
1187799452,32cyclictest1999-21ssh11:00:131
1188599443,36cyclictest9799-21apt-get08:15:122
1186699442,37cyclictest1841-21sh09:30:140
1186699441,39cyclictest18310-21ssh10:15:130
1188599438,26cyclictest3844-21sh09:11:412
1188599431,36cyclictest15837-21apt-get08:55:182
1189399422,24cyclictest35966irq/58-eth010:25:133
1188599423,35cyclictest0-21swapper/209:15:152
1188599421,37cyclictest0-21swapper/210:40:142
1188599421,37cyclictest0-21swapper/209:45:142
1188599421,29cyclictest30648-21ssh10:03:092
1188599421,23cyclictest0-21swapper/211:45:142
1188599421,23cyclictest0-21swapper/211:45:132
1188599420,28cyclictest19027-21ssh12:39:102
11877994226,7cyclictest131rcu_preempt12:35:121
1186699421,35cyclictest7343-21ssh12:15:100
1188599411,36cyclictest19857-21apt-get09:40:132
1188599411,36cyclictest0-21swapper/210:10:142
1187799414,29cyclictest16133-21ssh09:20:141
1187799412,27cyclictest18263-21ssh11:45:131
1187799412,27cyclictest18263-21ssh11:45:121
1186699417,5cyclictest131rcu_preempt11:50:120
11866994112,12cyclictest4741-21ssh09:49:340
11866994112,11cyclictest23046-21ssh10:52:320
1188599401,32cyclictest14489-21apt-get09:55:142
1188599401,23cyclictest0-21swapper/211:58:572
1188599401,22cyclictest0-21swapper/210:20:562
1188599401,22cyclictest0-21swapper/210:20:552
1187799401,29cyclictest30827-21ssh12:10:141
11866994028,8cyclictest14609-21apt-get08:20:140
11866994028,8cyclictest14609-21apt-get08:20:130
11866994014,16cyclictest14047-21grep09:35:200
11866994012,11cyclictest27705-21ssh11:13:510
1188599391,34cyclictest0-21swapper/211:25:122
1188599390,36cyclictest9716-21ssh10:26:402
1187799395,5cyclictest131rcu_preempt08:15:121
1187799391,31cyclictest25216-21ssh09:25:121
11866993914,10cyclictest21574-21ntpq07:50:250
11866993913,7cyclictest18754-21ssh11:08:530
11866993910,11cyclictest21372-21ssh12:20:360
11866993910,10cyclictest5577-21ssh10:06:100
11866993910,10cyclictest10403-21sh10:27:300
1188599383,30cyclictest0-21swapper/209:35:122
1188599381,34cyclictest0-21swapper/208:00:152
1188599381,33cyclictest5644-21ntp_192.168.11411:00:242
1188599380,20cyclictest11015-21apt-get08:50:142
11866993831,4cyclictest24229-21apt-get08:30:140
11866993828,6cyclictest5662-21apt-get11:20:150
11866993814,9cyclictest28288-21sed08:00:190
11866993814,9cyclictest25077-21apt-get12:25:020
11866993811,9cyclictest20942-21ssh12:04:260
1189399379,19cyclictest35966irq/58-eth010:05:123
11893993719,10cyclictest35966irq/58-eth011:55:133
1188599372,30cyclictest0-21swapper/210:55:132
1188599371,33cyclictest0-21swapper/208:20:122
1188599371,33cyclictest0-21swapper/208:20:122
1188599371,31cyclictest0-21swapper/210:45:132
1188599370,34cyclictest27765-21cut09:05:192
1188599370,24cyclictest2835-21ssh11:17:092
1188599370,21cyclictest14190-21users11:40:322
1186699379,12cyclictest3760-21cpuspeed07:35:140
1186699379,10cyclictest14884-21ssh09:19:030
1186699378,11cyclictest27700-21ssh09:44:020
11866993730,3cyclictest7479-21apt-get10:45:140
11866993724,10cyclictest25490-21apt-get10:55:110
11866993711,11cyclictest6669-21ssh12:14:380
11866993711,10cyclictest24978-21latency07:20:220
1188599361,31cyclictest0-21swapper/209:00:132
1188599361,20cyclictest5844-21ntp_192.168.11412:30:272
1188599361,15cyclictest0-21swapper/212:19:502
1188599360,23cyclictest26721-21sh10:19:482
1188599360,23cyclictest19504-21sshd10:32:362
1188599360,22cyclictest4960-21sendmail_mailqu08:40:272
11877993613,16cyclictest21-21ksoftirqd/112:05:121
1187799361,26cyclictest25500-21ssh10:55:121
1186699369,18cyclictest30688-21ssh10:03:130
1186699369,10cyclictest32713-21ssh09:28:390
1186699367,10cyclictest28707-21ssh11:30:450
11866993629,3cyclictest25484-21apt-get09:05:110
11866993627,5cyclictest16809-21apt-get07:15:130
11866993623,9cyclictest5509-21apt-get09:50:130
11866993623,7cyclictest0-21swapper/009:10:120
11866993612,10cyclictest9315-21ssh11:03:230
11866993611,13cyclictest30055-21diskmemload10:42:020
1188599351,31cyclictest20115-21grep09:20:282
1188599351,22cyclictest0-21swapper/212:06:202
1188599351,19cyclictest0-21swapper/207:40:122
1188599350,32cyclictest25435-21timerandwakeup07:20:282
1188599350,32cyclictest0-21swapper/208:10:122
1188599350,22cyclictest27171-21ssh11:50:072
11866993518,12cyclictest21838-21apt-get12:05:120
11866993511,9cyclictest14506-21ssh11:40:550
11866993511,8cyclictest17150-21ssh12:36:530
11866993511,6cyclictest26899-21ssh10:35:420
11866993511,10cyclictest20679-21ssh10:34:010
11866993510,9cyclictest16834-21open_inodes07:45:240
11866993510,10cyclictest10872-21ssh11:58:070
1189399342,21cyclictest35966irq/58-eth010:00:143
11885993415,16cyclictest0-21swapper/207:30:122
1188599341,30cyclictest0-21swapper/211:05:192
1188599341,25cyclictest0-21swapper/207:35:132
1188599341,21cyclictest24191-21ssh12:23:592
1188599340,31cyclictest0-21swapper/208:05:132
1188599340,19cyclictest0-21swapper/212:25:442
1187799343,23cyclictest664-21ntpd08:05:151
1186699349,8cyclictest6871-21ssh11:37:370
1186699349,11cyclictest15634-21ssh10:12:240
11866993428,2cyclictest20675-21apt-get09:00:140
11866993411,9cyclictest28992-21ssh11:15:080
11866993411,9cyclictest19612-21ssh09:55:530
11866993411,8cyclictest716-21ssh10:21:390
11866993411,8cyclictest716-21ssh10:21:390
11866993410,9cyclictest21468-21ssh09:21:030
1189399330,22cyclictest35966irq/58-eth009:00:133
1188599332,16cyclictest19417-21apt-get08:25:122
1188599331,29cyclictest4713-21ssh12:12:162
1188599331,29cyclictest19504-21sshd12:02:402
1188599331,28cyclictest21887-21apt-get10:35:132
1188599331,23cyclictest6389-21tune2fs08:45:162
1188599330,30cyclictest30021-21ssh11:32:182
1188599330,30cyclictest11983-21/usr/sbin/munin07:10:102
1188599330,29cyclictest73-21kswapd007:15:182
1188599330,29cyclictest30055-21diskmemload09:30:122
1188599330,24cyclictest16902-21/usr/sbin/munin07:45:252
1186699338,10cyclictest10956-21date08:50:010
11866993313,7cyclictest29524-21sed08:35:180
11866993313,7cyclictest1995-21irqrtprio08:40:190
1188599322,22cyclictest323-21timerwakeupswit08:35:282
1188599321,17cyclictest0-21swapper/209:27:492
1188599320,29cyclictest24322-21diskstats08:30:162
1188599320,29cyclictest22782-21ssh10:52:132
1186699328,8cyclictest4964-21apt-get08:10:120
11866993224,5cyclictest32575-21apt-get08:05:130
11866993220,6cyclictest19417-21apt-get08:25:130
11866993211,8cyclictest25319-21ssh11:48:090
11866993211,8cyclictest25319-21ssh11:48:080
11866993211,8cyclictest14324-21vmstat08:15:310
11866993210,8cyclictest7829-21ssh12:31:260
1188599311,22cyclictest30166-21/usr/sbin/munin07:25:302
1187799316,5cyclictest131rcu_preempt10:20:131
1187799316,5cyclictest131rcu_preempt10:20:121
11893993025,3cyclictest321rcuc/307:15:113
1188599300,26cyclictest11960-21latency_hist07:10:022
1187799302,18cyclictest21-21ksoftirqd/109:05:121
1186699304,10cyclictest27365-21grep07:25:200
11866993015,11cyclictest12097-21diskstats07:10:140
11866993010,8cyclictest9502-21kthreadcore07:40:190
11893992924,3cyclictest321rcuc/307:20:123
11877992916,10cyclictest25252-21cron09:05:011
1186699298,9cyclictest20234-21usage_EthcTask08:55:340
1186699296,6cyclictest0-21swapper/008:50:270
1187799285,3cyclictest131rcu_preempt08:50:141
1188599271,24cyclictest18136-21/usr/sbin/munin07:50:112
1187799272,20cyclictest1823-21apt-get09:30:141
11866992715,9cyclictest31389-21diskstats07:30:130
1187799264,13cyclictest12877-21ssh12:00:131
1187799262,18cyclictest30154-21ssh10:39:361
1187799261,3cyclictest131rcu_preempt09:50:151
1187799261,21cyclictest28933-21ssh09:45:141
1189399253,10cyclictest0-21swapper/312:33:423
1189399252,14cyclictest35966irq/58-eth011:50:123
11893992512,8cyclictest35966irq/58-eth012:25:143
1189399250,16cyclictest35966irq/58-eth008:20:133
1189399250,16cyclictest35966irq/58-eth008:20:133
1187799252,18cyclictest10820-21ssh09:35:121
11877992513,4cyclictest21-21ksoftirqd/111:58:061
1189399248,10cyclictest35966irq/58-eth012:20:143
11893992419,3cyclictest321rcuc/310:35:183
11893992419,3cyclictest321rcuc/309:35:113
11893992419,3cyclictest321rcuc/308:05:283
11893992418,4cyclictest321rcuc/311:11:453
1187799246,3cyclictest131rcu_preempt08:25:281
1187799243,16cyclictest0-21swapper/110:09:361
1187799241,9cyclictest16556-21ntp_kernel_err10:30:271
11877992412,3cyclictest21-21ksoftirqd/111:21:271
1186699245,9cyclictest0-21swapper/007:55:120
1189399233,16cyclictest35966irq/58-eth011:25:123
11893992317,4cyclictest321rcuc/311:18:003
11893992317,4cyclictest321rcuc/310:59:373
11893992317,4cyclictest321rcuc/307:50:143
11893992310,9cyclictest0-21swapper/311:44:003
1187799232,15cyclictest201rcuc/108:10:121
1187799232,13cyclictest630-21Async09:36:311
11877992314,6cyclictest9253-21ssh11:40:131
11877992312,7cyclictest0-21swapper/110:10:141
11877992310,4cyclictest21-21ksoftirqd/112:20:261
1189399222,16cyclictest35966irq/58-eth010:45:123
11893992217,3cyclictest321rcuc/311:07:553
11893992217,3cyclictest321rcuc/309:40:153
11893992214,5cyclictest0-21swapper/310:16:533
1187799221,6cyclictest131rcu_preempt10:18:561
11877992214,6cyclictest15747-21ssh09:20:011
1189399213,12cyclictest0-21swapper/312:38:433
1189399212,10cyclictest35966irq/58-eth007:10:123
11893992116,3cyclictest321rcuc/311:45:083
11893992116,3cyclictest321rcuc/311:45:083
11893992116,3cyclictest321rcuc/311:00:303
11893992116,3cyclictest321rcuc/310:35:013
11893992116,3cyclictest321rcuc/309:22:233
11893992115,4cyclictest321rcuc/311:20:353
11893992115,4cyclictest321rcuc/307:45:153
11893992113,5cyclictest0-21swapper/309:29:003
11893992113,5cyclictest0-21swapper/309:17:513
1187799219,5cyclictest21-21ksoftirqd/112:15:151
1187799217,8cyclictest21-21ksoftirqd/108:55:341
1187799217,8cyclictest21-21ksoftirqd/108:45:291
1187799214,10cyclictest21-21ksoftirqd/109:58:231
11877992114,4cyclictest14693-21cpu11:25:131
1187799211,3cyclictest131rcu_preempt08:35:021
11877992112,7cyclictest21-21ksoftirqd/112:25:141
11877992112,6cyclictest32702-21ssh11:35:131
11877992112,6cyclictest30925-21ssh10:40:141
11877992111,8cyclictest21-21ksoftirqd/110:45:301
1187799211,16cyclictest31294-21apt-get07:30:121
1187799211,16cyclictest20081-21ssh11:10:131
11877992111,3cyclictest21-21ksoftirqd/111:31:271
11877992110,3cyclictest21-21ksoftirqd/112:30:141
11866992110,8cyclictest11884-21sed07:10:010
1189399206,10cyclictest0-21swapper/312:09:413
1189399204,11cyclictest35966irq/58-eth009:50:283
1189399203,14cyclictest35966irq/58-eth012:15:103
1189399203,13cyclictest0-21swapper/310:10:283
11893992015,3cyclictest321rcuc/308:30:223
11893992015,3cyclictest321rcuc/307:35:263
11893992014,4cyclictest321rcuc/311:30:253
11893992014,4cyclictest321rcuc/309:31:393
11893992013,5cyclictest0-21swapper/309:58:323
11893992013,5cyclictest0-21swapper/309:11:553
1189399201,11cyclictest35966irq/58-eth008:50:153
11893992010,5cyclictest0-21swapper/307:25:143
1187799209,2cyclictest21-21ksoftirqd/110:03:361
1187799208,4cyclictest0-21swapper/107:40:241
1187799208,3cyclictest21-21ksoftirqd/107:50:241
1187799206,8cyclictest0-21swapper/111:05:351
1187799201,6cyclictest26280-21vmstat07:20:321
11877992012,6cyclictest16637-21ssh10:50:171
11877992010,2cyclictest21-21ksoftirqd/108:20:141
11877992010,2cyclictest21-21ksoftirqd/108:20:141
11893991913,4cyclictest0-21swapper/312:00:183
11893991913,4cyclictest0-21swapper/311:35:373
11893991912,5cyclictest0-21swapper/310:50:473
11893991912,5cyclictest0-21swapper/309:45:043
11893991912,4cyclictest0-21swapper/312:10:193
11893991910,6cyclictest0-21swapper/310:40:303
1187799196,10cyclictest27535-21cron08:00:021
1187799194,9cyclictest844-21munin-node08:00:141
11877991910,2cyclictest131rcu_preempt07:25:171
1189399182,10cyclictest0-21swapper/308:40:153
1189399182,10cyclictest0-21swapper/308:15:143
11893991812,4cyclictest0-21swapper/308:55:163
11893991811,4cyclictest321rcuc/308:45:213
1189399181,10cyclictest0-21swapper/307:40:113
1187799188,2cyclictest21-21ksoftirqd/110:29:401
1187799188,2cyclictest21-21ksoftirqd/107:45:221
1187799187,3cyclictest21-21ksoftirqd/107:10:011
11877991813,3cyclictest0-21swapper/107:10:211
11877991811,4cyclictest21-21ksoftirqd/108:35:221
1187799180,3cyclictest131rcu_preempt08:40:201
1189399172,10cyclictest0-21swapper/307:30:133
11893991712,3cyclictest321rcuc/308:35:283
11893991712,3cyclictest321rcuc/308:25:203
11893991711,4cyclictest0-21swapper/308:00:303
11893991710,4cyclictest0-21swapper/309:05:203
11877991712,3cyclictest0-21swapper/107:35:241
11893991611,3cyclictest0-21swapper/308:10:173
11893991611,3cyclictest0-21swapper/307:55:113
1189399131,10cyclictest0-21swapper/307:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional