You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-12 - 02:56

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Sun Apr 12, 2026 00:45:11)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
7610998045,23cyclictest28366irq/58-eth021:10:153
761099485,26cyclictest28366irq/58-eth000:30:163
7544994736,7cyclictest25056-21apt-get00:25:150
755899442,39cyclictest21107-21apt-get00:05:141
755899441,37cyclictest7122-21apt-get20:50:131
755899431,38cyclictest0-21swapper/120:45:151
755899431,35cyclictest10545-21apt-get20:20:111
7544994333,4cyclictest13968-21apt-get19:50:130
7610994227,8cyclictest28366irq/58-eth000:15:133
755899420,23cyclictest2307-21ssh21:14:281
754499421,5cyclictest131rcu_preempt22:15:130
7544994213,16cyclictest26207-21diskmemload23:46:460
754499421,28cyclictest21124-21ssh00:05:130
755899411,37cyclictest1058-21ssh22:25:121
755899411,35cyclictest0-21swapper/100:20:111
755899411,24cyclictest8965-21ssh21:52:551
758399401,24cyclictest32578-21ssh23:00:142
755899402,34cyclictest0-21swapper/120:00:111
755899401,35cyclictest12315-21apt-get21:20:131
755899401,34cyclictest31923-21apt-get19:35:131
755899401,34cyclictest31923-21apt-get19:35:121
755899401,34cyclictest0-21swapper/122:20:121
755899401,23cyclictest32109-21kthreadcore21:30:201
7544994010,10cyclictest8856-21ssh00:33:120
7610993930,6cyclictest28366irq/58-eth021:45:143
755899391,35cyclictest28441-21apt-get20:05:121
755899391,33cyclictest0-21swapper/120:30:141
755899390,23cyclictest5408-21apt-get20:15:021
755899390,22cyclictest13878-21ssh21:37:361
755899390,22cyclictest13878-21ssh21:37:361
7544993912,8cyclictest16172-21ssh21:55:410
7544993911,10cyclictest1958-21ssh23:17:580
7558993818,15cyclictest0-21swapper/123:15:141
755899381,33cyclictest0-21swapper/119:25:141
754499388,10cyclictest26967-21ssh22:38:510
7544993832,3cyclictest5723-21apt-get20:15:140
7544993813,9cyclictest11904-21cron21:20:010
7544993811,11cyclictest8382-21ssh23:56:120
7544993810,11cyclictest8171-21sed23:20:310
761099375,17cyclictest28366irq/58-eth022:10:133
761099374,18cyclictest28366irq/58-eth023:25:133
7583993723,8cyclictest22197-21apt-get23:30:142
755899372,32cyclictest17424-21apt-get19:20:111
755899372,31cyclictest12565-21apt-get19:15:121
755899372,31cyclictest0-21swapper/121:55:131
755899372,31cyclictest0-21swapper/100:25:131
755899371,31cyclictest0-21swapper/123:10:131
755899371,20cyclictest13001-21ssh22:49:101
755899370,33cyclictest32551-21apt-get23:00:131
755899361,32cyclictest17206-21apt-get23:45:131
755899361,24cyclictest18692-21memory20:25:231
755899360,23cyclictest26207-21diskmemload22:50:471
754499369,8cyclictest2664-21ssh00:10:330
754499369,15cyclictest1151-21df22:25:130
7544993611,11cyclictest11951-21acpi20:55:130
7544993611,11cyclictest11951-21acpi20:55:120
7583993515,13cyclictest531-21usb-storage19:10:112
7583993515,13cyclictest531-21usb-storage19:10:112
755899351,31cyclictest0-21swapper/119:30:111
754499359,8cyclictest23679-21ssh00:23:520
754499359,8cyclictest11503-21ssh22:11:100
754499358,10cyclictest28547-21rm22:04:320
754499357,10cyclictest15495-21cron00:20:010
7544993524,8cyclictest18330-21apt-get23:10:130
7544993524,7cyclictest25428-21apt-get21:45:140
7544993517,11cyclictest0-21swapper/022:20:120
7544993510,10cyclictest19921-21ssh23:27:460
758399341,20cyclictest29437-21ssh22:05:132
755899340,21cyclictest26207-21diskmemload23:26:081
754499349,11cyclictest14063-21ssh22:50:110
754499348,8cyclictest32056-21ssh23:51:360
754499346,11cyclictest16510-21ssh23:44:440
7544993411,9cyclictest8510-21rm23:04:340
761099333,17cyclictest28366irq/58-eth023:35:123
761099332,16cyclictest28366irq/58-eth022:55:133
7610993319,9cyclictest28366irq/58-eth022:00:133
7583993321,9cyclictest16330-21apt-get21:40:122
755899333,27cyclictest0-21swapper/121:05:131
755899331,29cyclictest2975-21awk23:55:161
755899331,28cyclictest31810-21ps23:35:181
755899331,28cyclictest23337-21apt-get22:55:131
754499339,10cyclictest10461-21/usr/sbin/munin20:50:220
754499337,11cyclictest22289-21/usr/sbin/munin19:55:270
7544993325,4cyclictest31400-21apt-get23:35:130
7544993320,10cyclictest2315-21apt-get20:45:140
7544993311,8cyclictest3107-21wc19:35:250
7544993311,8cyclictest3107-21wc19:35:250
7544993311,4cyclictest12445-21ssh21:20:160
7544993310,6cyclictest26207-21diskmemload00:39:250
755899321,28cyclictest28356-21apt-get22:40:121
755899321,28cyclictest19960-21/usr/sbin/munin22:35:191
755899321,28cyclictest0-21swapper/121:00:141
755899321,28cyclictest0-21swapper/100:35:131
755899321,27cyclictest3388-21ssh00:30:221
755899321,27cyclictest28958-21apt-get22:05:051
755899321,27cyclictest1518-21ps00:10:261
755899321,19cyclictest0-21swapper/121:16:111
754499327,9cyclictest19805-21sh00:04:020
754499327,8cyclictest31835-21rm22:59:450
7544993220,8cyclictest4316-21apt-get19:40:130
7544993211,7cyclictest28001-21ssh21:27:330
7610993112,8cyclictest28366irq/58-eth022:50:143
758399312,12cyclictest16813-21ssh00:36:542
758399311,19cyclictest7175-21ssh21:35:132
758399311,19cyclictest7175-21ssh21:35:132
755899311,27cyclictest2501-21awk23:55:011
755899311,27cyclictest2068-21sshd20:42:361
755899311,26cyclictest9663-21irqrtprio19:45:181
755899311,19cyclictest14241-21ssh23:05:351
755899311,16cyclictest20013-21ssh00:04:151
755899310,27cyclictest20175-21ssh22:00:091
755899310,20cyclictest22051-21ssh21:41:311
755899310,19cyclictest14379-21ssh23:42:111
755899310,18cyclictest28426-21ssh23:32:011
754499319,11cyclictest2737-21irqstats21:50:180
754499318,8cyclictest31557-21ssh21:10:380
754499318,7cyclictest0-21swapper/022:06:530
761099301,16cyclictest28366irq/58-eth021:40:123
7583993017,9cyclictest15381-21apt-get20:25:132
755899304,24cyclictest7820-21ntpq23:20:271
755899301,27cyclictest0-21swapper/122:30:141
755899301,25cyclictest0-21swapper/119:40:111
755899301,24cyclictest0-21swapper/119:55:131
755899300,27cyclictest4364-21processes20:10:271
754499307,7cyclictest17898-21ssh23:09:580
754499307,7cyclictest0-21swapper/020:20:150
754499306,7cyclictest13549-21ssh21:37:140
754499306,7cyclictest13549-21ssh21:37:130
7544993013,13cyclictest0-21swapper/023:30:130
758399296,11cyclictest27-21ksoftirqd/221:30:232
758399293,10cyclictest0-21swapper/200:19:542
7583992918,3cyclictest11127-21apt-get21:55:132
755899291,26cyclictest6804-21apt-get00:15:121
755899291,25cyclictest28557-21/usr/sbin/munin20:35:271
755899291,24cyclictest23963-21wc22:20:011
755899291,24cyclictest21401-21ssh21:25:131
755899290,25cyclictest30886-21ssh21:46:141
7544992917,10cyclictest3386-21ssh21:31:010
7544992914,11cyclictest0-21swapper/021:40:120
7544992910,7cyclictest6305-21kthreadcore22:45:200
7544992910,15cyclictest0-21swapper/020:05:120
7610992814,6cyclictest28366irq/58-eth022:05:143
758399289,15cyclictest27-21ksoftirqd/222:00:132
758399287,10cyclictest27-21ksoftirqd/223:24:452
758399286,8cyclictest27-21ksoftirqd/200:20:202
758399285,13cyclictest0-21swapper/220:05:122
7583992815,9cyclictest24381-21apt-get22:20:122
758399281,11cyclictest27-21ksoftirqd/200:10:152
755899281,24cyclictest6330-21fschecks_time22:10:181
754499288,9cyclictest17519-21/usr/sbin/munin19:20:140
754499287,7cyclictest0-21swapper/021:05:180
7544992821,4cyclictest29913-21apt-get20:40:130
7544992821,3cyclictest16782-21apt-get21:00:140
7544992816,9cyclictest28368-21ssh22:40:120
7544992816,8cyclictest19223-21/usr/sbin/munin20:25:290
7544992812,3cyclictest0-21swapper/019:15:270
761099275,13cyclictest28366irq/58-eth022:15:133
761099273,14cyclictest28366irq/58-eth023:10:133
7610992715,8cyclictest28366irq/58-eth021:30:143
758399273,15cyclictest27-21ksoftirqd/223:55:122
758399272,19cyclictest0-21swapper/220:10:142
758399271,18cyclictest27064-21ssh22:38:592
758399271,16cyclictest27-21ksoftirqd/221:19:392
758399271,16cyclictest27-21ksoftirqd/220:15:132
755899271,22cyclictest0-21swapper/120:55:131
755899271,22cyclictest0-21swapper/120:55:121
754499276,7cyclictest26207-21diskmemload22:30:390
7544992713,10cyclictest28831-21tail20:35:290
761099262,10cyclictest28366irq/58-eth020:45:143
758399266,5cyclictest27-21ksoftirqd/219:32:222
758399264,13cyclictest22346irq/40-dwc2_hso19:35:122
758399264,13cyclictest22346irq/40-dwc2_hso19:35:112
758399263,5cyclictest27-21ksoftirqd/223:40:032
758399262,4cyclictest17552-21fschecks_count19:20:142
7583992620,3cyclictest21402-21apt-get21:25:142
7583992616,8cyclictest261rcuc/223:17:092
7583992612,10cyclictest0-21swapper/222:15:242
758399261,20cyclictest0-21swapper/221:10:142
755899261,21cyclictest13953-21find19:50:121
7544992616,7cyclictest23621-21apt-get20:00:120
7544992615,8cyclictest23695-21open_inodes20:30:280
7544992615,7cyclictest1205-21sed20:10:190
7544992614,9cyclictest9164-21grep19:45:140
7544992614,9cyclictest26124-21unixbench_singl19:25:290
761099257,8cyclictest28366irq/58-eth022:25:103
761099251,14cyclictest28366irq/58-eth000:25:133
758399256,6cyclictest27-21ksoftirqd/223:29:592
758399256,11cyclictest27-21ksoftirqd/223:13:452
758399255,5cyclictest27-21ksoftirqd/200:00:292
758399253,4cyclictest27-21ksoftirqd/222:25:352
758399252,10cyclictest2852-21sendmail_mailst23:35:272
758399251,6cyclictest27-21ksoftirqd/200:25:452
758399251,6cyclictest26874-21ssh00:06:352
7583992516,3cyclictest261rcuc/223:45:502
758399251,15cyclictest26155-21ssh23:50:042
755899251,21cyclictest7755-21apt-get19:10:111
755899251,21cyclictest7755-21apt-get19:10:111
7544992515,7cyclictest7836-21df19:10:140
7544992515,7cyclictest7836-21df19:10:130
758399246,7cyclictest27-21ksoftirqd/200:30:152
758399246,11cyclictest27-21ksoftirqd/222:49:122
758399244,5cyclictest26207-21diskmemload21:20:202
758399243,13cyclictest22346irq/40-dwc2_hso19:15:112
758399242,14cyclictest27-21ksoftirqd/222:41:112
7583992418,2cyclictest22258-21apt-get19:25:132
7583992414,3cyclictest261rcuc/222:56:442
758399241,20cyclictest0-21swapper/220:00:112
758399241,16cyclictest22346irq/40-dwc2_hso19:50:122
758399241,15cyclictest27-21ksoftirqd/222:50:202
761099237,11cyclictest0-21swapper/300:24:543
758399234,6cyclictest27-21ksoftirqd/220:40:272
758399233,5cyclictest20566-21cat21:00:292
758399232,5cyclictest27-21ksoftirqd/222:34:422
7583992311,9cyclictest7480-21ssh21:51:122
758399231,17cyclictest6117-21ssh22:10:142
7583992310,10cyclictest15148-21cron20:25:002
758399230,5cyclictest12046-21df20:55:152
758399230,5cyclictest12046-21df20:55:152
755899231,20cyclictest7529-21cyclictest19:10:021
7544992316,2cyclictest27058-21apt-get19:30:110
7544992312,8cyclictest0-21swapper/019:10:030
761099229,8cyclictest0-21swapper/323:20:143
761099222,12cyclictest28366irq/58-eth000:00:133
761099221,14cyclictest28366irq/58-eth019:50:133
758399227,10cyclictest0-21swapper/221:45:132
758399225,9cyclictest131rcu_preempt23:06:412
758399225,10cyclictest27-21ksoftirqd/220:51:282
758399221,17cyclictest20484-21kthreadcore19:55:222
758399221,14cyclictest0-21swapper/220:49:152
758399221,11cyclictest0-21swapper/219:40:442
758399221,10cyclictest0-21swapper/220:30:352
758399220,4cyclictest271ksoftirqd/220:35:312
761099218,8cyclictest28366irq/58-eth021:55:143
7610992113,5cyclictest0-21swapper/322:42:303
7610992112,4cyclictest0-21swapper/320:40:123
758399211,10cyclictest0-21swapper/221:07:462
758399210,13cyclictest27-21ksoftirqd/219:45:142
761099209,6cyclictest0-21swapper/322:21:593
7610992014,4cyclictest0-21swapper/323:55:303
7610992014,4cyclictest0-21swapper/321:35:473
7610992014,4cyclictest0-21swapper/321:35:473
7610992014,4cyclictest0-21swapper/300:05:133
7610992011,4cyclictest0-21swapper/300:39:083
761099201,12cyclictest0-21swapper/319:55:293
761099199,8cyclictest0-21swapper/321:16:433
761099198,8cyclictest0-21swapper/322:47:473
761099192,11cyclictest0-21swapper/323:45:053
761099192,10cyclictest0-21swapper/319:10:133
761099192,10cyclictest0-21swapper/319:10:133
7610991913,4cyclictest0-21swapper/323:50:123
7610991913,4cyclictest0-21swapper/323:17:433
7610991913,4cyclictest0-21swapper/322:35:213
7610991913,4cyclictest0-21swapper/322:30:323
7610991913,4cyclictest0-21swapper/321:51:353
7610991912,5cyclictest0-21swapper/323:05:113
7610991912,5cyclictest0-21swapper/320:35:123
761099191,12cyclictest0-21swapper/320:10:283
761099191,12cyclictest0-21swapper/319:40:163
7610991910,4cyclictest0-21swapper/323:01:563
758399191,10cyclictest7664-21ls19:10:022
761099182,10cyclictest0-21swapper/320:20:123
761099182,10cyclictest0-21swapper/320:15:053
761099182,10cyclictest0-21swapper/319:25:143
761099182,10cyclictest0-21swapper/319:15:123
7610991812,4cyclictest0-21swapper/323:40:253
7610991812,4cyclictest0-21swapper/323:30:163
7610991812,4cyclictest0-21swapper/321:25:283
7610991812,4cyclictest0-21swapper/321:20:223
7610991812,4cyclictest0-21swapper/300:10:133
7610991810,3cyclictest0-21swapper/320:30:203
761099172,9cyclictest0-21swapper/319:30:123
761099172,10cyclictest0-21swapper/320:50:143
761099172,10cyclictest0-21swapper/320:00:143
761099172,10cyclictest0-21swapper/319:45:133
761099162,11cyclictest0-21swapper/321:10:013
761099161,8cyclictest0-21swapper/319:35:103
761099161,8cyclictest0-21swapper/319:35:103
761099151,8cyclictest0-21swapper/320:05:283
761099151,10cyclictest0-21swapper/321:00:143
7610991510,3cyclictest0-21swapper/320:55:283
7610991510,3cyclictest0-21swapper/320:55:283
7610991510,3cyclictest0-21swapper/320:25:163
761099122,6cyclictest0-21swapper/319:20:123
761099112,6cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional